Validity of Scandura and Ragins' (1993) Multidimensional Mentoring Measure: An Evaluation and Refinement

Transcription

1 University of Miami Scholarly Repository Management Faculty Articles and Papers Management Validity of Scandura and Ragins' (1993) Multidimensional Mentoring Measure: An Evaluation and Refinement Stephanie L. Castro Florida Atlantic University Terri A. Scandura PhD University of Miami, Ethlyn A. Williams Florida Atlantic University Recommended Citation Castro, Stephanie L.; Scandura, Terri A. PhD; and Williams, Ethlyn A., "Validity of Scandura and Ragins' (1993) Multidimensional Mentoring Measure: An Evaluation and Refinement" (2004). Management Faculty Articles and Papers This Conference Proceeding is brought to you for free and open access by the Management at Scholarly Repository. It has been accepted for inclusion in Management Faculty Articles and Papers by an authorized administrator of Scholarly Repository. For more information, please contact

2 Multidimensional Mentoring Measure 1 Validity of Scandura and Ragins (1993) Multidimensional Mentoring Measure: An Evaluation and Refinement Stephanie L. Castro Department of Management College of Business Florida Atlantic University 2912 College Avenue Davie, FL (954) (954) fax scastro@fau.edu Terri A. Scandura Department of Management School of Business University of Miami 414 Jenkins Building Coral Gables, FL (305) (305) fax scandura@miami.edu Ethlyn A. Williams Department of Management College of Business Florida Atlantic University 777 Glades Road Boca Raton, FL (561) (561) fax ewilliam@fau.edu A previous version of this paper was presented at the Southern Management Association Meetings, San Antonio, Texas, We would like to thank Chet Schriesheim for his constructive comments on an earlier draft.

3 Multidimensional Mentoring Measure 2 Validity of Scandura and Ragins (1993) Multidimensional Mentoring Measure: An Evaluation and Refinement Abstract The establishment of a mentoring relationship can be important to an individual s career for multiple reasons. However, in order to study this construct, we must be able to accurately measure it. In this paper, three separate studies were conducted to examine and refine Scandura and Ragins (1993) multidimensional mentoring measure. In Study 1, an empirical assessment of the content validity of the measure was conducted. The convergent and discriminant validity, reliability, and item-total correlations were then examined in Study 2, and the measure was reduced to nine items. The convergent and discriminant validity, reliability, and item-total correlations of this reduced measure were then assessed. In Study 3, these same evaluations were repeated. Concurrent validity was also evaluated by relating the measure to organizational outcomes. Based on these studies, initial indications of the validity of the revised 9-item measure are good. Strengths, limitations, and directions for future research are discussed. (149 words) Keywords: Measurement design, reliability and validity, survey research, construct validity, mentoring

4 Multidimensional Mentoring Measure 3 Mentoring relationships have been discussed in the academic and practitioner communities as critical for the development of protégés, as these relationships generally have a positive impact on individual and organizational outcomes. A recent meta-analysis demonstrated that protégés experience substantial career benefits, including positive objective outcomes such as increased compensation and promotions as well as subjective outcomes such as increased job satisfaction and reduced turnover intentions (Allen, Eby, Poteet, Lentz, & Lima, 2004). However, the field is still in its relative infancy, as evidenced by the small number of studies included in the meta analysis (i.e., 43). As we move forward and attempt to improve our understanding of the nature of the mentoring relationship (Underhill, 2006), we need to be concerned with the validity of the instruments we use to measure the mentoring construct. The importance of obtaining evidence supporting the construct validity of measures used in empirical research has been emphasized by many (e.g., Nunnally & Bernstein, 1994; Schwab, 1980). Unfortunately, researchers all too often do not heed this advice, and use measures in organizational studies that lack sufficient evidence of construct validity (Cook, Hepworth, Wall, & Warr, 1981; Podsakoff & Dalton, 1987; Schriesheim, Powers, Scandura, Gardiner, & Lankau, 1993; Stone-Romero, 1994). This violates the basic principles of scientific research, for without accurate instruments to measure constructs, we can have little confidence in the results. As Korman (1974, p. 194) pointed out, The point is not that accurate measurement is nice. It is necessary, crucial, etc. Without it we have nothing. In this study we evaluate the validity of Scandura and Ragins (1993) multidimensional mentoring measure. The measure has 15 items, and is comprised of three scales measuring three dimensions: career support, psychosocial support, and role modeling. Although this measure has been used in a number of studies, there is limited evidence of its validity. The majority of the

5 Multidimensional Mentoring Measure 4 validity-related evidence to date addresses concurrent validity. Thus far, Scandura and Ragin s mentoring measure and/or its subdimensions have been positively related to personal learning and job satisfaction (Lankau & Scandura, 2002), value similarity and supportiveness (Neilson, Carlson, & Lankau, 2001), promotion rate and salary level (Scandura, 1992), and feminity (Scandura & Ragins, 1993). The measure and/or its subdimensions have been negatively associated with role ambiguity and intent to leave (Lankau & Scandura, 2002), family interference with work (Nielson et al., 2001), and organizational rank and mentorship experience (Scandura & Ragins, 1993). Minimal support for the factor structure exists. Scandura and Ragins (1993) found support for the three factor structure in their initial study. Williams (1999) also found support for three factors. Arguably, given the limited evidence thus far, more information regarding the validity of Scandura and Ragins (1993) measure is needed. Thus, the primary purpose of this paper is to evaluate and refine Scandura and Ragin s (1993) mentoring measure through a comprehensive, programmatic approach. Quantitative, empirical analyses of content validity, convergent validity, and discriminant validity are conducted, and reliability and item-total correlations are examined. Based on our results, a 9- item version of Scandura and Ragins (1993) measure is proposed, and concurrent validity is then assessed. The 9-item measure is then evaluated in a separate confirmatory sample. Study 1 Study 1 was conducted to empirically assess the content validity of Scandura and Ragins (1993) mentoring measure. The quantitative methodology described by Schriesheim et al. (1993) was employed. The mentoring items and the three corresponding subdimension definitions were presented to judges, as well as a fourth category, defined as Behaviors which clearly do not fall

6 Multidimensional Mentoring Measure 5 into the other categories. The judges were asked to evaluate the items consonance with the theoretical definitions. Method Sample Students attending a medium-sized private university in the southeastern United States were used as judges. The use of students for this type of analysis should not be problematic, since they are not asked about actual work or mentoring experience; they are only asked to make semantic judgments about item fidelity to definitions (cf. Schriesheim et al., 1993). The total sample size was 169, and 50% were male. The average age was 24 years old and 96% were employed or had been previously employed. The sample was 40% Caucasian, 31% Hispanic, 10% African American, 6% Asian, and 12% other. Measure Scandura and Ragin s (1993) measure, comprised of three scales measuring career support, psychosocial support, and role modeling, was employed. Judges were given the 15 items, the three subdimension definitions, and the none of the above category. (A copy of the form is available from the authors upon request.) Analyses The data from the content adequacy assessment was analyzed using principal axis factor analysis with oblique rotations (delta = 0). The data were analyzed in two ways: (1) factors with an eigenvalue greater than or equal to 1 were extracted, and (2) the number of factors was specified (based on theory). For this latter procedure, three factors were specified (one each for career support, psychosocial support, and role modeling).

7 Multidimensional Mentoring Measure 6 Results The results for the two analyses (extracting factors with eigenvalues greater than or equal to 1.0 or specifying the number of factors) were identical. The first four unrotated eigenvalues for the Scandura and Ragins (1993) 15-item measure were 3.45, 2.55, 1.89, and.86, supporting the extraction of three factors. The total variance explained was 41%. The factor loadings (pattern coefficients) are presented in Table 1. Following the recommendation of Ford, MacCallum, and Tait (1986), loadings of.40 or greater on the intended theoretical factor were considered meaningful for interpretation, and cross-loadings of.30 or greater were considered excessive. All items loaded.40 or stronger on the intended factor, and there were no crossloadings of.30 or greater Insert Table 1 about here Discussion The results of the factor analysis strongly support the content validity of Scandura and Ragins (1993) measure. All of the items exceeded the criteria set forth by Ford et al. (1986). Importantly, there were no cross loadings, indicating that the items are unidimensional. Additionally, the fourth category, None of the above, was not supported. Thus, the items can be considered to strongly reflect their intended theoretical constructs. Study 2 The purpose Study 2 was to assess scale reliability, concurrent validity, and convergent and discriminant validity. Concurrent validity was evaluated by relating the mentoring measure and subscales to job satisfaction, organizational commitment, and anxiety. The two former constructs were chosen as a significant positive correlation with mentoring is expected, based on

8 Multidimensional Mentoring Measure 7 theory and previous empirical analyses (e.g., Allen et al., 2004). General anxiety was also measured, but was expected to have only a weak (negative) relationship with mentoring. Convergent validity was evaluated by examining (1) the ability of the hypothesized factor structure of each scale to account for a large amount of covariation among the measure s items, (2) the statistical significance and size of the item loadings on the hypothesized factors, (3) the proportion of variance in the latent variables accounted for by the indicators using Fornell and Larcker s (1981) average variance indicator (ρ vc(η) ), and (4) the correlations with Ragins and McFarlin s (1990) mentoring measure. Discriminant validity was assessed by (1) testing that each of the correlations among the scales was significantly less than 1, and (2) comparing ρ vc(η) with scale intercorrelations. Fornell and Larcker (1981) argued that to demonstrate discriminant validity, the average variance in a latent construct s indicators should be greater than the square of the construct s correlations with other latent constructs. Due to the somewhat disappointing outcomes of the Study 2 analyses, the scale was reduced to a 9-item measure, and the analyses repeated. Method Sample Employed MBA students attending classes at a medium-sized private university in the southeastern United States were surveyed. The total sample size was 474; of these, 255 (54%) indicated having had a mentor at some time in their careers. Only those who had been mentored were used in the analyses. Listwise deletion of missing data resulted in a sample of N = 246. This sample was used in the Lisrel analyses, evaluating convergent and discriminant validity, and the reliability and item-total analyses. However, for the analyses including the Ragins and McFarlin (1990) measure, missing data resulted in a sample of N = 160. This smaller sample

9 Multidimensional Mentoring Measure 8 was only used in the regression analyses comparing the two measures (reported in Table 8) and the correlations (presented in Table 9). In the mentored sample, 54.7% were male, and the average age was 28.8 years old. The sample was 52% Caucasian, 7.4% African American, 23% Hispanic, 6.3% Asian, and 3.1% indicated other. On average, the respondents had 4.8 years of experience prior to joining their current organization, and 3.3 years experience in their current position. Respondents jobs ranged from first line supervisor to CEO. Measures Scandura and Ragins (1993) measure was used. The coefficient alpha reliability estimate for this administration of the original 15-item measure was.93, while the reliability of the scores on the revised 9-item measure was.91 and the reliability of the scores on the 3-item revised scales ranged from.82 to.85. The 33-item measure developed by Ragins and McFarlin (1990) was used as an alternative measure of the mentoring construct. The measure is comprised of eleven scales, each with three items: sponsorship, coaching, protection, challenge, exposure, friendship, social, parent, role modeling, counseling, and acceptance-and-confirmation. The social and parent scales were developed for specific mentoring situations (cross-gender), and are not directly comparable to the Scandura and Ragins (1993) scale. Thus, the six items measuring these two dimensions were eliminated from the analyses in this study. 1 The coefficient alpha for the data collected on the 27-item scale was.91. Job satisfaction was measured using the 20-item short form of the Minnesota Satisfaction Questionnaire (MSQ; Weiss, Dawis, England, & Lofquist, 1967). The MSQ was employed due to its careful development and refinement, as well as the considerable evidence which supports

10 Multidimensional Mentoring Measure 9 its validity and reliability (e.g., Wanous, 1974; Weiss et al., 1967). The coefficient alpha reliability estimate in this sample was.86. The Organizational Commitment Questionnaire (OCQ; Porter, Steers, Mowday, & Boulian, 1974) was used to measure organizational commitment. There have been numerous studies which have investigated the construct validity of the OCQ, and while there is evidence supporting its validity, there are also concerns. In particular, some have questioned the factor structure of the measure and recommended a 9-item form (e.g., Mathieu & Zajac, 1990). However, the 9-item version of the OCQ only measures affective commitment. Since our desire was to capture the entire theoretical construct of organizational commitment, we used the complete OCQ in these analyses. Coefficient alpha in this sample was.85. Anxiety was measured by the State-Trait Anxiety Inventory (Speilberger, Gorush & Lushene, 1970). Respondents were asked to indicate how they generally felt. Sample items include I feel nervous, and I am worried. This measure has been used in over 150 studies, and substantial evidence supports its validity (e.g., Rule & Traver, 1983; Sherwood & Westerback, 1983). The coefficient alpha reliability estimate in this data was.90. Analyses Convergent and discriminant validity. The convergent and discriminant validity of the measure was evaluated using Lisrel 8.51 (Jöreskog & Sörbom, 1993). Sample covariances served as input for the maximum likelihood estimations, each item was specified as loading on only one factor, the errors were specified as uncorrelated among themselves, and the latent trait factor correlations were freely estimated. Four rival models were tested. A single factor model with all the items loading on one global mentoring factor was tested. Two alternate two factor models were tested. In the first two

11 Multidimensional Mentoring Measure 10 factor model (Model 2a), the role modeling and career support items loaded on one factor while the psychosocial support items loaded on a second factor. In the second two factor model (Model 2b), the role modeling and psychosocial support items loaded on one factor while the career support items loaded on a second. 2 Finally, a three factor model was tested, with the career, psychosocial, and role modeling items each loading on their respective factors. Scale reliabilities and item-total correlations. Coefficient alpha internal consistency scale reliabilities and item-total correlations were examined for each measure and its scales. Reduced 9-item measure. Based on the item analyses, scale reliabilities, content validity results, and factor analysis results, Scandura and Ragins (1993) measure was reduced to nine items, three items per scale. Scale reliabilities, item-total correlations, and convergent and discriminant validity were then assessed on the reduced measure and scales. Comparison of the two mentoring measures. An additional analysis used to support the construct validity of the reduced 9-item mentoring measure was an investigation of the ability of the Ragins and McFarlin (1990) measure to add to explained variance in outcome measures over the 9-item measure. Since both measures assess the same global construct (mentoring), neither should add explanatory power over the other. As Liden and Maslyn (1998) did comparing the LMX-7 scale (Scandura & Graen, 1984) with their newly-constructed LMX-MDM scale, we ran a series of hierarchical regression analyses. Using job satisfaction, organizational commitment, and anxiety as dependent variables (each in separate regression analyses), the revised 9-item measure was entered first. In the second step, the 27-item Ragins and McFarlin (1990) scale was added. The change in R 2 was then evaluated to determine whether the Ragins and McFarlin (1990) measure was able to explain any additional variance in the outcome variables. The analyses were then repeated, reversing the order of entry.

12 Multidimensional Mentoring Measure 11 Concurrent validity. Concurrent validity was examined by looking at correlations between the mentoring measure and its three scales and outcome variables (job satisfaction, organizational commitment, and anxiety). Convergent and discriminant validity Results The fit statistics for the four alternative models are presented in Table 2. The hypothesized three factor model fit best, and significantly better than either of the two factor models. The chi-square difference between the three factor model and the next-best fitting model (model 2a) was (df = 2), significant at the p.001 level. However, the chi-square statistic was statistically significant (χ 2 = , df = 87) and the ratio of the chi-square to degrees of freedom approached 3.5, indicating only a moderate fit (Bentler & Bonett, 1980). The fit indices also indicated a moderate fit, with the SRMSR =.067, the GFI =.84, the TLI =.88, the NFI =.86, and the CFI =.90. While the fit indices for the three factor model were better than the alternative models, only the CFI met the.90 standard (Medsker, Williams, & Holahan, 1994). The root mean square error of approximation (RMSEA) for the three factor model was.11, indicating a poor fit (Browne & Cudeck, 1993). Additionally, model comparisons were made using the 90% confidence interval of the RMSEA. Given that the confidence interval for the three factor model overlaps with the confidence interval for model 2a, we cannot conclude that the three factor model has a significantly better fit (Browne & Cudeck, 1993) Insert Table 2 about here The completely standardized factor loadings and error variances for the 15-item measure are presented in Table 3. All of the loadings were statistically significant (p.05), with an

13 Multidimensional Mentoring Measure 12 average factor loading of.74. However, there were several factor loadings that were relatively low. The loadings for item 3 (.69), item 11 (.63), and 14 (.58) were all below.70. The disattenuated correlations and average variances are presented in Table 4. The disattenuated correlations ranged from.74 to.83, with an average of.77. Although the correlations were high, they were significantly less than 1.0 (p.01). Additionally, they were not high enough to suggest a second order factor existed. Finally, in looking at the average variance (ρ vc(η) ), the hypothesized dimensions (latent constructs) accounted for a large proportion of variance (on average) in their indicators. The average variance accounted for ranged from.52 to.58, with an average of.55. In comparing ρ vc(η) to the disattenuated correlations, three of the six comparisons did not fit the criteria. The ρ vc(η) for the career support scale was.52, which was less than the square of the disattenuated correlation between career support and psychosocial support (.55), and less than the square of the disattenuated correlation between career support and role modeling (.69). Also, the ρ vc(η) for the role modeling scale was.56, which was less than the square of the disattenuated correlation between career support and role modeling (.69) Insert Tables 3 and 4 about here Finally, the correlation between the 15-item measure and Ragins and Mc Farlin s (1990) 27-item measure (excluding the social and parent scales) was r =.73 (p.01; see Table 9). Based on these results, there appeared to be some evidence of the convergent and discriminant validity of Scandura and Ragins (1993) 15-item measure. However, the moderate overall model fit, the poor model comparison results, and the average variance comparisons indicated that the measure could be improved.

14 Multidimensional Mentoring Measure 13 Scale Reliabilities and Item-total Correlations The coefficient alpha internal consistency reliability estimate for the 15-item measure was.93. Item-total correlations were moderately strong, generally ranging from.60 to.78, with two items below.60 (for item 11, r it =.55, and for item 14, r it =.49). The reliability estimates for the data collected for the career support, psychosocial support, and role modeling scales were.88,.87, and.83, respectively. Item analyses indicated that deleting single items would not increase the scale reliabilities for either the full measure or the three scales. Revised 9-item Measure Based on the results of the analyses, we felt the 15-item Scandura and Ragins (1993) measure needed to be modified. The measure was reduced to nine items, hereinafter referred to as the Mentoring Functions Questionnaire, or MFQ-9. Three items were retained for each dimension (career support, psychosocial support, and role modeling the retained items are listed in Table 6). The theoretical construct definitions were taken into consideration in determining which items to retain, as were the item-total correlations, factor loadings, and content adequacy results. The rationale behind the retention/elimination of particular items is discussed below. For the career support scale, rather than retain items that focused on one of the particular subdimensions (e.g., coaching), we chose to retain items that were more general in nature. For example, we dropped the item My mentor advised me of promotional opportunities because the situation may preclude the availability of such opportunities (i.e., there may be no promotional opportunities in the organization). The lack of promotional opportunities, however, does not prevent a mentor from providing career support in other ways. The other items eliminated were similar, in that the mentor may or may not have had the opportunity to engage in

15 Multidimensional Mentoring Measure 14 the behavior due to circumstances. Notably, the item with the low loading from the confirmatory factor analysis (item 3) was specific (rather than general), and thus it was eliminated. For the psychosocial support scale, we eliminated items that were possibly biased in that they could only apply to mentor-protégé relationships occurring at the same physical location. The items I often go to lunch with my mentor and I socialize with my mentor after work were dropped, as it may be physically impossible to engage in these behaviors (different physical location, different working hours, inability to take off at the same time, etc.), yet psychosocial support could still occur. Additionally, the former item obtained a low confirmatory factor loading, further contributing to the reasoning for elimination. Finally, for the role modeling scale, we eliminated the one item that was not behavioral. Item 14 (which also had a low loading in the confirmatory factor analysis), I respect my mentor s knowledge of the profession, was not related to any behavior on the part of the mentor, and thus was not something the protégé could model. The three items retained referred to things that could be imitated (i.e., motivating others, teaching others), or to the actual act of trying to imitate the mentor s behavior. The analyses (confirmatory factor analyses, reliabilities, and item-total correlations) were then repeated on the revised 9-item measure, or the MFQ-9. Convergent and discriminant validity. The four alternative models were tested using the MFQ-9, and the fit statistics are presented in Table 5. The hypothesized three factor model fit best. The chi-square statistic was statistically significant (χ 2 = 79.3, df = 24, p.001), and the ratio of the chi-square to degrees of freedom was approximately 3, indicating a moderate fit (Bentler & Bonett, 1980). The three factor model fit significantly better than the next-best fitting model (2a), with a chi-square difference of (df = 2), significant at the p.001 level. The

16 Multidimensional Mentoring Measure 15 fit indices indicated an excellent fit, with the SRMSR =.046, GFI =.92, TLI =.93, NFI =.94, and CFI =.95. (We recognize that since this is the sample used to modify the measure, a good fit was expected.) However, as with the 15-item measure, the RMSEA was.11 (above the.10 recommendation; Browne and Cudeck, 1993), and the RMSEA confidence interval overlapped with model 2a Insert Table 5 about here The completely standardized factor loadings and error variances for the MFQ-9 are presented in Table 6. All of the loadings were statistically significant (p.01), and most were large, ranging from.69 to.89 with an average factor loading of.79. Disattenuated correlations and average variance indices are presented in Table 7. Correlations ranged from.76 to.80, with an average of.78. The correlations were significant (p.01) and significantly less than 1.0 (p.01). Although high, the correlations between the subdimensions were not high enough to suggest a second order factor. Finally, the hypothesized dimensions accounted for a large proportion of variance (on average) in their indicators. Specifically, ρ vc(η) =.58 for career support, ρ vc(η) =.65 for psychosocial support, and ρ vc(η) =.63 for role modeling. Comparing ρ vc(η) with the disattenuated correlations, four of the six comparisons fit the criteria, and the other two were close. The ρ vc(η) for the career support scale was.58, which was slightly less than the square of the disattenuated correlation between career support and role modeling (.61). Additionally, ρ vc(η) for the role modeling scale was.63, which was less than the square of the disattenuated correlation between psychosocial support and role modeling (.64).

17 Multidimensional Mentoring Measure Insert Tables 6 and 7 about here The correlation between the MFQ-9 and Ragins and Mc Farlin s (1990) 27-item measure (excluding the social and parent scales) was r =.73. This correlation is identical to the correlation between the 15-item measure and Ragins and Mc Farlin s (1990) 27-item measure. Reliability and item-total correlations. The coefficient alpha internal consistency reliability for the scores on the MFQ-9 was.91. The reliabilities for the scores on the career support scale, psychosocial support scale, and role modeling scale were.82,.85, and.83. Itemtotal correlations for all three scales items ranged from.62 to.78. Item analyses indicated that the deletion of any single item would not result in improvement in these reliabilities. Comparison of Ragins and McFarlin s Measure and the Revised 9-Item Measure Using hierarchical regression, the ability of the two measures to explain variance in outcomes over the other was evaluated (cf. Liden & Maslyn, 1998). The results are presented in Table 8. The addition of either measure to equations with the other did not explain statistically significant increases in variance in the outcome measures. Assessment of Concurrent Validity Insert Table 8 about here Table 9 presents the means, standard deviations, and intercorrelations for the measures. The 15-item Scandura and Ragins measure and the MFQ-9 were highly correlated (r =.94), and outcome variable correlations with the MFQ-9 were very similar to those with the 15-item measure. The MFQ-9 was significantly related to job satisfaction (r =.23, p.01) and marginally significantly related to organizational commitment (r =.14, p.10). The career

18 Multidimensional Mentoring Measure 17 support and role modeling dimensions were significantly related to satisfaction (r =.20 and.26, p.05 and p.01, respectively) and commitment (r =.13 and.19, p.10 and p.05, respectively). Psychosocial support was significantly related to satisfaction (r =.15, p.10) but not to commitment (r =.06). None of the three scales were significantly related to anxiety. (Parenthetically, the statistical significance of the correlations between the MFQ-9 and the three outcome variables are identical to the statistical significance of the same correlations for the Ragins and McFarlin [1990] measure.) Insert Table 9 about here Discussion The initial findings of Study 2 offered only moderate support for the validity of the 15- item measure. Using the construct definitions as well as the empirical results of Studies 1 and 2, we revised the measure to 9 items and reevaluated it, repeating the analyses. Substantial support was obtained for the MFQ-9. However, this is not surprising, since the same data used to revise the measure were used to reevaluate it. A stronger test comes from an additional, confirmatory sample. Hence, we conducted Study 3. Study 3 This study was conducted to evaluate the validity of the MFQ-9. Convergent and discriminant validity, reliabilities, and item total correlations were evaluated as they were in Study 2.

19 Multidimensional Mentoring Measure 18 Method Sample This sample was drawn from employed CPAs, and a total of 1024 surveys were completed. Of these, 795 indicated having had a mentor at some point in their careers, and thus these comprised the study sample. Sixty-eight percent of the mentored sample was male, and the average age was 30. Twelve percent of the sample was African American, and 88% were Hispanic or Caucasian. Average organizational tenure was 2.3 years, ranging from a few months to 28 years. Measures The MFQ-9, drawn from Scandura and Ragins (1993), was used. The reliability for this data on the MFQ-9 was.78, and the reliabilities were.77,.67, and.69 for the psychosocial support, career support, and role modeling scale data, respectively. Analyses Convergent and discriminant validity. The convergent and discriminant validity of the measure was evaluated using maximum likelihood estimation in Lisrel 8.51 (Jöreskog & Sörbom, 1993). Sample covariances served as input for the estimations, each item was specified as loading on only one factor, the errors were specified as uncorrelated among themselves, and the latent trait factor correlations were freely estimated. The same four rival models tested in Study 2 were tested here, and the same examination of convergent and discriminant validity was conducted. Scale reliabilities and item-total correlations. Coefficient alpha internal consistency reliabilities and item-total correlations were examined for the MFQ-9 and its scales.

20 Multidimensional Mentoring Measure 19 Results Convergent and Discriminant Validity The fit statistics for the four alternative models are presented in Table 5. The hypothesized three factor model fit best, with χ 2 = and df = 24 (p.01), and a ratio of the chi-square to degrees of freedom of approximately 2, indicating a good fit (Bentler & Bonett, 1980). Additionally, the three factor model fit significantly better than the next-best fitting model (2a), with a chi-square difference of (df = 2), significant at the p.001 level. Finally, the fit indices indicated an excellent fit: SRMSR =.031, GFI =.98, TLI =.96, NFI =.96, CFI =.98, and RMSEA =.04. The RMSEA confidence interval comparisons show that the three factor model is significantly better than any of the others (i.e., there is no overlap with any of the other models confidence intervals). The completely standardized factor loadings and error variances for the MFQ-9 are presented in Table 6. All of the loadings were statistically significant (p.01), and most were large. The loadings ranged from.45 to.81. Four loadings were below.70: item 1 (.68), item 5 (.45), item 12 (.58), and item 15 (.67). Disattenuated correlations and average variance indices are presented in Table 7. Correlations ranged from.42 to.54, with an average of.49. The correlations were significant (p.01) and significantly less than 1.0 (p.01). The correlations between the subdimensions were not high enough to suggest a second order factor. Finally, the hypothesized dimensions accounted for a large proportion of variance (on average) in their indicators: ρ vc(η) =.44 for career support, ρ vc(η) =.55 for psychosocial support, and ρ vc(η) =.44 for role modeling. Comparing ρ vc(η) with the disattenuated correlations, all twelve comparisons fit Fornell and

21 Multidimensional Mentoring Measure 20 Larcker s (1981) criteria (i.e., the average variance was greater than the square of the construct s correlations with other variables). Reliability and Item-total Correlations The reliability estimate for the data on the MFQ-9 was.78 (reliabilities were.67,.77, and.69 for career, psychosocial, and role modeling scale data, respectively). Item-total correlations for all three scales items ranged from.38 to.66. Only the deletion of item 5 (r it =.38) in the career support scale would result in an improvement in the coefficient alpha (from.67 to.70). Discussion While the construct validity evidence collected in Study 3 was generally supportive, some was not. The fit statistics obtained for the theoretical model indicated an excellent fit of the model to the data. Notably, both model comparison techniques (change in χ 2 and RMSEA confidence interval comparisons) supported the three factor model as significantly better than the alternative models. Additionally, all of the factor loadings were statistically significant. Evidence for discriminant validity was also strong. Each of the factor correlations was significantly less than 1.0, and the average variance comparisons using ρ vc(η) met the criteria in all of the comparisons. On the other hand, some of the evidence was not so supportive. Four of the factor loadings in this study were below the.70 threshold, albeit most only slightly so. Reliability for the overall measure was adequate, but two of the scales were slightly below the.70 rule-ofthumb. While these results are somewhat surprising, they should not generate too much alarm. First, it is important to recall that all of the items demonstrated strong content adequacy (Study 1). Additionally, the low loadings were not replicated in Study 2--only item 1 had low loadings

22 Multidimensional Mentoring Measure 21 in both Study 2 and Study 3. Finally, it should be noted that the loadings for item 1 were close to.70 (.69 and.68 in Studies 2 and 3). The results of this study draw attention to item 5. This item was the only one for which deletion would have resulted in an improvement in a scale s reliability. Item 5 also obtained the lowest factor loading in this study. Given these results, item 5 should probably be looked at more carefully in future studies. However, we note that the poor results obtained in this study are possibly sample specific, as item 5 was not problematic in Study 2. General Discussion Unfortunately, as noted by many others (e.g., Cook et al., 1981; Podsakoff & Dalton, 1987; Schriesheim et al., 1993; Stone-Romero, 1994), the field generally does not devote enough attention to measurement development and validation. Researchers tend to use measures to evaluate substantive hypotheses prior to evaluating the construct validity of the measures. Thus, the purpose of this study was to evaluate the validity of a commonly used measure in mentoring research, Scandura and Ragins (1993) mentoring measure. The analyses of Scandura and Ragins (1993) 15-item scale in Study 2 were not as supportive as expected. While the content validity analysis was excellent (three factors, strong theoretical factor loadings, no substantial cross-loadings), the results of some of the other analyses were disappointing. The confirmatory factor model did not fit well, in that three of the four fit indices employed were below the.90 criterion. Additionally, the comparisons recommended by Fornell and Larcker (1981) were not supportive of the distinctiveness of the constructs. The poor results of the confirmatory factor analysis and the Fornell and Larcker (1981) average variance comparisons suggested that the Scandura and Ragins measure could be modified to better represent the construct. The content adequacy results implied that the current

23 Multidimensional Mentoring Measure 22 items did indeed represent the theoretical constructs, and thus that new items were probably not needed. Using the results of the above analyses and the theoretical construct definitions, the MFQ-9 was constructed and reevaluated. The analyses of the MFQ-9 were very supportive of its validity. Convergent and discriminant analyses were strongly supportive of the MFQ-9. In the confirmatory factor analyses, the three factor model fit the data extremely well in both Study 2 and Study 3. Factor loadings were strong in both samples, with some minor exceptions in Study 3. Similarly, reliabilities and item-total correlations were strong in Study 2. While two of the estimated reliabilities in Study 3 were below the.70 recommendation (Nunnally & Bernstein, 1994), they were only slightly below. These lower factor loadings and reliability estimates could be due to the sample (accountants), or to the diverse activities represented by these items since it is a short form attempting to capture broad behavioral categories. Additional evidence of the convergent and discriminant validity strongly supported the MFQ-9. The average variance explained by each latent variable was considerable. Furthermore, the comparisons of the squared correlations with the average variance indicator (Fornell & Larcker, 1981) all exceeded the criteria in Study 3, supporting the distinctiveness of the factors. The correlation between the 15-item scale and the MFQ-9 scale was very high (r =.94), indicative that these two measures assess the same global construct. The hierarchical regression results also supported the convergent validity of the MFQ-9. It appears to assess the same global mentoring construct as Ragins and McFarlin s measure. Based on these results, we consider the convergent and discriminant validity of the MFQ-9 to be strongly supported. Furthermore, the concurrent validity assessment also supported the MFQ-9. All of the relationships were as predicted, with one nonsignificant exception (psychosocial support and commitment).

24 Multidimensional Mentoring Measure 23 A recent study by Pellegrini and Scandura (2005) provided additional support for the MFQ-9. A three factor structure was supported in two separate samples (those satisfied with their mentors and those dissatisfied with their mentors). Interestingly, the psychometric properties of the MFQ-9 were stronger in the dissatisfied sample than in the satisfied sample. The authors recommended that further investigation is needed. Limitations The primary limitation of this study was the use of self-report measures, which may lead to method bias. However, as Allen et al. s (2004) recent meta-analysis noted, affective variables have stronger correlations with mentoring functions than objective variables (e.g., compensation, promotions, salary growth). With the use of affective variables (as in this study), it is necessary to assess employees perceptions directly. Additionally, the inclusion of the anxiety measure and the differential correlations it obtained with the mentoring scales (as opposed to the positive correlations with job satisfaction and organizational commitment) help to alleviate some of the concerns of common method bias. That is, it does not appear that respondents were merely responding positively to each item. A second limitation of this study is the sample in Study 3 used to confirm the factor structure of the revised Scandura and Ragins (1993) measure. This sample, comprised entirely of CPAs, could be considered unique and the results from analyses potentially ungeneralizable to other organizational populations. However, the Study 3 sample was merely used to confirm results obtained with a much more diverse sample (Study 2). Relatedly, it would have been ideal to evaluate concurrent validity by looking at the relationship of the MFQ-9 with outcome variables in Study 3. Unfortunately, we were unable to collect this additional data due to organizational constraints.

25 Multidimensional Mentoring Measure 24 Future Research Obviously, given the results of Studies 2 and 3, more research is needed on the validity of the MFQ-9. In particular, investigations of relationships with other variables in other organizational settings would be useful. Also, further investigation of item 5 is required. Additionally, the measurement model tests conducted in Study 2 point to a potentially interesting area for further study. Traditionally, role modeling is considered part of the psychosocial function (Kram, 1985). Scandura s measure (Scandura, 1992; Scandura and Ragins, 1993) proposes that the role modeling function is separate and distinct from the psychosocial function. While the analyses in Study 3 supported this distinction, the analyses in Study 2 did not. What is particularly interesting is that the Study 2 analyses indicated that role modeling was more closely associated with the career support function (not the psychosocial function). This presents a potentially useful area for future study, both from a theoretical and an empirical standpoint. Summary The analyses reported in this paper indicated that the original 15-item multidimensional mentoring measure (Scandura & Ragins, 1993) needed refinement. Based on the results of multiple samples and analyses, the measure was refined to a 9-item measure assessing three dimensions: career support, psychological support, and role modeling. This shorter measure still provides researchers the ability to look at either the three dimensions separately, or at the global mentoring construct. Ultimately, a shorter measure with superior evidence of validity had been developed: the 9-item Mentoring Functions Questionnaire (MFQ-9). The MFQ-9 is now recommended (instead of the original 15-item version developed by Scandura and Ragins [1993]).

26 Multidimensional Mentoring Measure 25 Footnote 1 The analyses were also conducted using the full 33-item measure, and the results are not substantially different from the analyses with the 27-item measure. The results are available from the first author upon request. 2 The third logical two factor model (with career support and psychosocial support loading on one factor and role modeling on the second) was not tested due to theoretical considerations. Kram (1985) hypothesized two dimensions, career support and psychosocial support, with role modeling a subdimension of psychosocial support. Yet the correlation in the Study 2 data indicated that role modeling was more strongly related to career support. Thus, Models 2a and 2b were tested to allow evaluation of whether role modeling is better viewed as a subdimension of career support or psychosocial support. Comparison with the three factor model then allowed determination of whether two or three factors best represented the data.

27 Multidimensional Mentoring Measure 26 References Allen, T.D., Eby, L.T., Poteet, M.L., Lentz, E., & Lima, L. (2004). Career benefits associated with mentoring for protégés: A meta-analysis. Journal of Applied Psychology, 89, Bentler, P.M., & Bonett, D.G. (1980). Significance tests and goodness-of-fit in the analysis of covariance structures. Psychological Bulletin, 88, Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp ). Thousand Oaks, CA: Sage. Cook, J.D., Hepworth, S.J., Wall, T.D., & Warr, P.B. (1981). The experience of work. London: Academic Press. Ensher, E.A., & Murphy, S.E. (1997). Effects of race, gender, perceived similarity, and contact on mentor relationships. Journal of Vocational Behavior, 50, Ford, J.K., MacCallum, R.C., & Tait, M. (1986). The application of exploratory factor analysis in applied psychology: A critical review and analysis. Personnel Psychology, 39, Fornell, C., & Larcker, D.F. (1981). Evaluating structural equation models with unobservable variables and measurement error. Journal of Marketing Research, 18, Jöreskog, K.G., & Sörbom, D. (1993). LISREL 8 user's reference guide. Chicago: Scientific Software International. Korman, A.K. (1974). Contingency approaches to leadership: An overview. In J. G. Hunt & L. L. Larson (Eds.), Contingency approaches to leadership (pp ). Carbondale: Southern Illinois University Press

28 Multidimensional Mentoring Measure 27 Kram, K.E. (1985). Mentoring at work: Developmental relationships in organizational life. Glenview, IL: Scott Foresman. Lankau, M.J., & Scandura, T.A. (2002). An investigation of personal learning in mentoring relationships: Content, antecedents and consequences. Academy of Management Journal, 45, Liden, R.C., & Maslyn, J.M. (1998). Multidimensionality of leader-member exchange: An empirical assessment through scale development. Journal of Management, 24, Mathieu, J., & Zajac, D.M. (1990). A review and meta-analysis of the antecedents, correlates, and consequences of organizational commitment. Psychological Bulletin, 108, Medsker, G.J., Williams, L., & Holahan. (1994). A review of current practices for evaluating causal models in organizational behavior and human resources management research. Journal of Management, 20, Neilson, T.R., Carlson, D.S., & Lankau, M.J. (2001). The supportive mentor as a means of reducing work-family conflict. Journal of Vocational Behavior, 59, Noe, R.A. (1988). An investigation of the determinants of successful assigned mentoring relationships. Personnel Psychology, 41, Nunnally, J.C., & Bernstein, I.H. (1994). Psychometric theory (3 rd edition). New York: McGraw-Hill. Pellegrini, E.K., & Scandura, T.A. (2005). Construct equivalence across groups: An unexplored issue in mentoring research. Educational and Psychological Measurement, 65,

29 Multidimensional Mentoring Measure 28 Podsakoff, P.M., & Dalton, D.R. (1987). Research methodology in organizational studies. Journal of Management, 13, Porter, L.W., Steers, R.M., Mowday, R.T., & Boulian, P.V. (1974). Organizational commitment, job satisfaction, and turnover among psychiatric technicians. Journal of Applied Psychology, 59, Ragins, B.R., & Cotton, J.L. (1999). Mentor functions and outcomes: A comparison o men and women in formal and informal mentoring relationships. Journal of Applied Psychology, 84, Ragins, B.R., & McFarlin, D.B. (1990). Perception of mentor roles in cross-gender mentoring relationships. Journal of Vocational Behavior, 37, Rule, W.R., & Traver, M.D. (1983). Test-retest reliabilities of the State-Trait Anxiety Inventory in a stressful social analogue situation. Journal of Personality Assessment, 47, Scandura, T.A. (1992). Mentorship and career mobility: An empirical investigation. Journal of Organizational Behavior, 13, Scandura, T.A., & Ragins, B.R. (1993). The effects of sex and gender role orientation on mentorship in male-dominated occupations. Journal of Vocational Behavior, 43, Schriesheim, C.A., Powers, K.J., Scandura, T.A., Gardiner, C.C., & Lankau, M.J. (1993). Improving construct measurement in management research: Comments and a quantitative approach for assessing the theoretical content adequacy of paper and pencil survey-type instruments. Journal of Management, 19,

30 Multidimensional Mentoring Measure 29 Schwab, D.P. (1980). Construct validity in organizational behavior. In B. Staw & L. Cummings (Eds.), Research in organizational behavior (Vol. 2, pp. 3-43). Greenwich CT: JAI Press. Sherwood, R.D., & Westerback, M.E. (1983). A factor analytic study of the State-Trait Anxiety Inventory utilized with preservice elementary teachers. Journal of Research in Science and Teaching, 20, Speilberger, C.D., Gorush, R.L. & Lushene, R.E. (1970). Manual for the state-trait anxiety inventory. Palo Alto, CA: Consulting Psychologists Press. Stone-Romero, E.F. (1994). Construct validity issues in organizational behavior. In J. Greenberg (Ed), Organizational behavior: The state of the science (pp ). Hillsdale, NJ: Erlbaum. Tepper, K., Shaffer, B.C., & Tepper, B.J. (1996). Latent structure of mentoring function scales. Educational and Psychological Measurement, 56, Underhill, C.M. (2006). The effectiveness of mentoring programs in corporate settings: A meta-analytical review of the literature. Journal of Vocational Behavior, 68, Wanous, J.P. (1974). A causal-correlational analysis of the job satisfaction and performance relationship. Journal of Applied Psychology, 59, Weiss, D.J., Dawis, R.V., England, G.W., & Lofquist, L.H. (1967). Manual for the Minnesota Satisfaction Questionnaire. Industrial Relations Center, University of Minnesota. Williams, E.A. (1999). A field experimental investigation of relationship-based mentoring training in team settings and team characteristics: Effects on employee attitudes and outcomes. Unpublished Doctoral Dissertation, University of Miami.