Specificity Matters: Criterion-Related Validity of Contextualized and Facet Measures of Conscientiousness in Predicting College Student Performance

Transcription

1 Journal of Personality Assessment, 0(0), 1 9, 2015 Copyright Ó Taylor & Francis Group, LLC ISSN: print / online DOI: / Specificity Matters: Criterion-Related Validity of Contextualized and Facet Measures of Conscientiousness in Predicting College Student Performance SANG EUN WOO, 1 JING JIN, 2 AND JAMES M. LEBRETON 3 1 Department of Psychological Sciences, Purdue University 2 Development Dimensions International, Bridgeville, Pennsylvania 3 Department of Psychology, Pennsylvania State University To enhance the predictive validity of self-report personality measures, 2 distinct ways of increasing specificity of personality measures have been proposed in the literature contextual specificity (i.e., providing a contextual referent) and content specificity (i.e., focusing on more specific constructs such as the Big Five facets). This study extends this line of research by examining whether there is an optimal way to configure, align, or integrate contextual and content specificity using measures of conscientiousness to predict college student success. A sample of 478 undergraduate students completed 4 measures of conscientiousness that varied in the level of content and contextual specificity. These forms of specificity were crossed to yield 4 distinct measures of conscientiousness. We then evaluated and compared the relative importance and the incremental importance of these different measures in the prediction of academic success. Superior predictive validity was found for both contextualized and facet measures of conscientiousness compared to a measure of global conscientiousness in predicting grade-point average and a broader behavioral criterion of student performance. When contextual and content specificity approaches were compared and combined, we observed the strongest predictive validity when the level of specificity is appropriately matched between predictor and criterion. Researchers have adopted a number of strategies to improve the predictive validity of self-report personality measures, including contextualizing items by placing them in a specific frame of reference such as school or work (e.g., Werner & Pervin, 1986) and focusing on narrower, facet-level traits (e.g., Paunonen, 1998). These two approaches are similar in that they both involve the increased level of specificity. However, they can be differentiated by the nature of specificity: The former involves contextual specificity, and the latter entails content specificity of personality predictors. Contextualizing personality items increases the specificity in measurement by attaching particular situations or role identities to how the trait is manifested in daily life (e.g., being conscientious at work vs. at school vs. at home), whereas a facet-level measure of personality does so by representing a smaller, relatively homogeneous cluster of behavioral inclinations (e.g., being organized vs. responsible vs. achievement-oriented). Studies suggest that contextualized measures of personality have superior predictive capabilities over generic, noncontextualized measures (e.g., Shaffer & Postlethwaite, 2012), and facet-level measures predict behavioral outcomes better than broad dimensional measures (e.g., Ashton, 1998; Dudley, Orvis, Lebiecki, & Cortina, 2006). What remains unclear is whether there is a predictive advantage in using contextualized measures over facet-level measures, or vice versa; whether contextualized facet measures would have superior predictive power over noncontextualized facets or contextualized broad factors (or dimensions); and how such predictive advantages might Received February 11, 2013; Revised November 12, Address correspondence to Sang Eun Woo, Department of Psychological Sciences, Purdue University, 703 Third Street, West Lafayette, IN 47907; sewoo@purdue.edu differ as a function of the criterion specificity (i.e., the breadth of behavioral space to be predicted). In view of this, this article seeks to add clarity to the question of whether there is an optimal way to configure or align the specificity of personality measurement. We address this issue by comparing alternative measures of the trait of conscientiousness in the prediction of college student success. Based on a sample of undergraduate students in the United States, we evaluate and compare the relative importance and the incremental importance of the following four types of conscientiousness measures: (a) a standard measure of the global trait (i.e., noncontextualized measure of conscientiousness), (b) a contextualized measure of the global trait (i.e., a measure of conscientiousness with the at school qualifier used to contextualize items), (c) a standard measure of a facet of conscientiousness (i.e., a noncontextualized measure of the achievement-striving subcomponent of conscientiousness), and (d) a contextualized measure of a facet (i.e., a measure of achievement-striving with the at school qualifier used to contextualize the items). To explore how such predictive advantages might differ by the breadth of criterion measures, we consider two types of criterion measures with differing levels of behavioral specificity: grade-point average (GPA; focusing on academic achievement), and a second measure that was designed to measure a broader range of intellectual, interpersonal, and intrapersonal learning and development outcomes. Among the dimensions of the Five-factor model (FFM), conscientiousness has been recognized as the most robust personality predictor for student performance (Poropat, 2009). Research shows that those with high conscientiousness are usually willing to exert more effort in attending classes and doing homework, and they have clearer goals; they also tend to be motivated to excel, organized, self-disciplined, and 1

2 2 WOO, JIN, LEBRETON responsible, all of which lead to higher educational achievements (Chamorro- Premuzic & Furnham, 2003; O Connor & Paunonen, 2007). Although meta-analytic findings reveal that broad measures of conscientiousness are reasonably effective predictors of student performance, the magnitude of effect sizes varies substantially across studies (Poropat, 2009). One explanation for such variability lies in the representation of situational factors in the wording of inventory items (Werner & Pervin, 1986, p. 622): Items that are situated in a specific context tend to have better predictive and criterion-related validities in predicting performance criteria in that context (Hunthausen, Truxillo, Bauer, & Hammer, 2003). Another probable explanation is that different measures of the global conscientiousness trait are not sampling items from the construct domain in identical ways. Specifically, it is commonly accepted that conscientiousness represents an omnibus trait that is made up of several subtraits or facets (e.g., achievement-striving, dutifulness, orderliness). If different measures differentially sample across these facets, it is likely that the resulting differences in predictive validities are due, at least in part, to differences in the level of content specificity matching (Paunonen, 1998). In the remainder of this section, we elaborate on these two measurement considerations (i.e., contextualization and facets) from the perspective of how one could maximize the criterion-related validity of personality measures in situations such as predicting academic performance and success. Theoretical and empirical research suggests that putting a specific context to personality items can improve criterionrelated validity of personality measures (Davison & Bing, 2009; Heggestad & Gordon, 2008; Hunthausen et al., 2003). Researchers have offered two theoretical explanations as to why the contextualization of personality measures might lead to higher criterion-related validity. The traditional explanation is that there could be subgroups among test-takers: Some refer to a certain frame of reference across all generic items, whereas others use a different frame of reference in responding to the same set of items. Adding a frame of reference could thus reduce between-person variability in responding to generic personality measures (Schmit, Ryan, Stierwalt, & Powell, 1995). Another explanation posits that adding a frame of reference might also help increase measurement reliability by mitigating within-person inconsistency, which can occur when respondents switch their frames of reference during the process of completing the survey, using one frame of reference to answer a set of items and another for a different set (Lievens, De Corte, & Schollaert, 2008). This frame-of-reference notion is grounded in the cognitive-affective system theory of personality (Mischel & Shoda, 1995), along with a number of other theories in social and personality psychology, which could be collectively characterized by a fundamental view that personality is shaped by (or accessed through) a dynamic interplay between dispositional and situational factors, and that certain behaviors and attitudes could be evoked, facilitated, or impeded by contextual cues and constraints (e.g., social roles or interpersonal relations; Bandura, 2001; Bem & Funder, 1978; Cervone, 2005; Funder, 2001; Mischel, 2004). Therefore, specifying situational contexts will likely reduce the range of behavioral expectations given a person s dispositional tendency, leading to more accurate predictions in linking personality to behavior in a certain context. Supporting these ideas, empirical studies have repeatedly shown that contextualized measures of personality yield superior predictive capabilities over generic, noncontextualized versions of personality measures via the frame-of-reference effect (e.g., Bing, Whanger, Davison, & Van Hook, 2004; Holtz, Ployhart, & Dominguez, 2005; Hunthausen et al., 2003; Robie, Schmit, Ryan, & Zickar, 2000; Schmit et al., 1995; Shaffer & Postlethwaite, 2012; D. B. Smith, Hanges, & Dickson, 2001). In particular, existing research suggests that school-specific personality items tend to predict educational outcomes better than (and have incremental validity over) generic, noncontextualized measures of personality (e.g., Bing et al., 2004; Lievens et al., 2008; Schmit et al., 1995). Therefore, we anticipated that a contextualized measure of conscientiousness would be a relatively better predictor of college student performance than a generic measure of conscientiousness, and it would explain variance in the criterion above and beyond the generic measure. Hypothesis 1: A contextualized measure of conscientiousness will demonstrate greater relative importance in the prediction of college student performance when compared to a global measure of conscientiousness. Hypothesis 2: A contextualized measure of conscientiousness will demonstrate incremental importance, above and beyond the global measure of conscientiousness in the prediction of college student performance. Another line of research has shown that facets, defined as narrower (or lower level) personality traits representing a relatively homogeneous cluster of behavioral tendencies (Paunonen & Ashton, 2001), are found to have superior criterionrelated validities compared to the broad dimensions of the FFM. Although FFM dimensions cover a broad range of behavioral domains, facets of these dimensions are conceptualized to represent a smaller number of behaviors that are more coherent with one another (Hampson, John, & Goldberg, 1986). Paunonen (1998) argued that there are substantial benefits to be derived from the measurement of components at all levels of the personality hierarchy, beyond simply the top level, which includes both the understanding and the prediction of behavior (p. 540). This statement is corroborated by the increasing volume of evidence that facets of a broad personality dimension often show differential relationships with various behavioral criteria (e.g., Roberts, Chernyshenko, Stark, & Goldberg, 2005), that facets provide as good or better prediction of behavior (e.g., Ashton, 1998), and that facets have incremental validity above and beyond broader personality factors in predicting many behavioral outcomes (e.g., Dudley et al., 2006). Several different taxonomies of personality facets have been proposed. One of the most popular is a taxonomy put forth by Costa and McCrae (1992), which was developed through a rational, theoretical process and was closely reflected in their NEO Revised Personality Inventory (NEO PI R) facet measures. Among the six NEO PI R facets of Conscientiousness (competence, order, dutifulness, achievement-striving, self-discipline, and deliberation), achievementstriving was selected as the focal facet predictor of college student performance for the purpose of our study. Individuals with high scores on achievement-striving are likely to work

3 CONTEXTUALIZATION VS. FACETS 3 hard and take their work seriously, set goals and turn them into action, require high standards, and have ambition to excel. As such, achievement-driven individuals are likely to be more engaged in academic pursuits and willing to put more time and effort into learning-related activities. Research has consistently shown that achievement-striving has one of the most robust relationships with academic achievement in college (Gray & Watson, 2002; Noftle & Robins, 2007; Schmit et al., 1995). It is also important to note that other facets of Conscientiousness, such as self-discipline, have been also closely associated with academic performance; achievement-striving was selected as an example of such facets rather than the only one that should be considered relevant. Regarding the predictive advantages of the achievement-striving facet specifically, we hypothesize as follows: Hypothesis 3: The achievement-striving facet will demonstrate greater relative importance in the prediction of college student performance when compared to the global measure of conscientiousness. Hypothesis 4: The achievement-striving facet will demonstrate incremental validity, above and beyond the global measure of conscientiousness in predicting college student performance. The contextualization approach improves criterion-related validity of personality measures by increasing contextual specificity, whereas the facet approach does so by increasing content specificity. Thus, an inevitable question arises: Would combining the two approaches (i.e., using contextualized measures of facets) yield further predictive advantages? In view of the nonredundancy in the nature of measurement specificity increased by contextualization and facet approaches (i.e., context vs. content specificity), we hypothesize the following: Hypothesis 5: A contextualized measure of the achievementstriving facet will demonstrate greater relative importance in the prediction of college student performance when compared to a global measure of conscientiousness, a contextualized measure of conscientiousness, and a noncontextualized measure of the achievement-striving facet in predicting college student performance. Hypothesis 6: A contextualized measure of the achievement-striving facet will demonstrate incremental importance in the prediction of student performance, above and beyond a global measure of conscientiousness, a contextualized global measure of conscientiousness, and a noncontextualized measure of the achievement-striving facet. Furthermore, we are also interested in comparing the relative efficacy of the contextualization approach and the facet approach in predicting behavioral outcomes. To our knowledge, no empirical research has compared the relative or incremental importance of these two approaches to specificity. Thus, we pose the following two exploratory questions without specifying directional hypotheses. Research Question 1: What is the relative importance of a contextualized measure of conscientiousness and a facet measure of conscientiousness (achievement-striving) in predicting college student performance? Research Question 2: Does a contextualized measure of conscientiousness add a significant amount of variance explained in college student performance above and beyond a facet measure of conscientiousness (achievement-striving), and vice versa? Although previous research provides a great deal of theoretical and empirical basis for the current investigations (e.g., Schmit et al., 1995), no empirical research to our knowledge has examined both relative importance and incremental validities of facet and contextualized measures within a single study. Relative importance tells how each predictor contributes to the overall criterion variance explained, considering the predictor s unique contribution as well as its contribution when combined with other (typically correlated) predictors. Incremental validity coefficients add another useful piece of information: They describe how much additional unique variance in the criterion is explained by adding a particular predictor to a regression equation containing other predictor variables. Relative importance is suggested to be superior to the evaluation of standardized regression weights in that it addresses concerns associated with correlated predictors (Darlington, 1968; Johnson & LeBreton, 2004), and provides complementary information about the overall contribution of predictors when used in conjunction with an incremental validity analysis (LeBreton, Hargis, Griepentrog, Oswald, & Ployhart, 2007). Finally, our study adds to the literature by examining these issues using two types of criterion measures with differing levels of specificity. In so doing, we explore how the specificity matching between predictors and criteria might play a role in maximizing the predictive power of personality measures. METHOD Sample A total of 478 undergraduates from two large Midwestern universities participated in the study by completing an online questionnaire and obtained course credit for their participation. The sample consisted of 263 women (55.0%), 213 men (44.6%), and 2 others who did not report their gender. There were 309 (64.6%) Whites, 96 (20.1%) Asians, 36 (7.5%) Blacks or African Americans, 16 (3.3%) Hispanics or Latinos, and 20 (4.2%) others. The age of the participants ranged from 18 to 39, with an average age of years (SD D 1.77). The survey included measures of conscientiousness and college student performance presented in the order described next (except that GPA was asked for at the beginning of the survey along with the demographic information). Measures Conscientiousness. All conscientiousness-related items included in the study are presented in the Appendix. Each item was rated on a 5-point scale ranging from 1 (strongly disagree) to5(strongly agree). Psychometric properties of each instrument are well-documented by the measures authors.

4 4 WOO, JIN, LEBRETON Conscientiousness as a broad personality dimension was assessed with 10 self-descriptive phrases obtained from the NEO PI R Conscientiousness domain measure of the International Personality Item Pool (IPIP; Goldberg et al., 2006), which is a noncommercialized version of the conscientiousness dimension as conceptualized and measured by Costa and McCrae (1992). The achievement-striving facet of Conscientiousness was measured using the 10-item scale from IPIP (Goldberg et al., 2006), which was specifically designed to assess the NEO PI R facet of achievement-striving. Items of the general measure of conscientiousness and the facet-level measure of achievement-striving did not overlap except for one item: Do just enough work to get by. For contextualized measures, we developed school-specific measures of conscientiousness by modifying the aforementioned measures of conscientiousness and achievementstriving from IPIP. Following the examples from previous frame-of-reference studies (Bing et al., 2004; Lievens et al., 2008), we added the tag at school to most of the items, and further revised some of the items to make the meanings clearer as needed (see the Appendix for the entire list of items). In addition, contextualized instructions were given to distinguish them from uncontextualized items as the following: The following statements are very similar to the ones you just saw in the previous page. The only difference is that these statements describe how a person is in the specific context of school work (e.g., studying, doing homework assignments). College student performance. To assess college student performance, we included two measures: (a) students self-report cumulative GPA during college, and (b) the 12- item Behaviorally Anchored Rating Scale (BARS) of college performance, which was developed and validated by Oswald, Schmitt, Kim, Ramsay, and Gillespie (2004). The BARS was designed to capture a wider range of college student performance dimensions, including intellectual (knowledge, learning, and artistic), interpersonal (multicultural, leadership, interpersonal, and citizenship), and intrapersonal (health, career, adaptability, perseverance, and ethics) dimensions. For each BARS item, a dimension name and its definition, as well as two examples of college-related critical incidents and example behaviors of low, average, and high levels of performance were presented. Students were asked to rate their level of performance for each dimension on a 7-point scale ranging from very low to very high (for more detailed information about the measure, see Oswald et al., 2004, pp. 191, ). Factor analytic results provided by Oswald et al. (2004) showed that the 12 BARS items represented one overall dimension. Our own confirmatory factor analysis also suggested there was a single, general factor; however, the overall model fit was less than ideal: x 2 (54, N D 478) D , p <.01; comparative fit index D.90; Tucker Lewis Index D.87; root mean square error of approximation D.09; standardized roont mean square residual D.05. Nevertheless, given the results of Oswald et al. and our results, we felt comfortable proceeding with a composite average score of the 12 BARS items as our indicator for overall college success. Coefficient alpha for the 12 BARS ratings was.79. Data Analysis 1 To evaluate H1, H3, and H5 (relative importance), we used a relative weight analysis (RWA; for technical and computational details, see Johnson, 2000). RWA has been shown to yield estimates of relative importance that are superior to traditional statistics such as zero-order correlations (r JY ) and standardized least-squares regression coefficient (b J ; Johnson & LeBreton, 2004; Krasikova, LeBreton, & Tonidandel, 2011). Specifically, RWA yields relative weights (RW J ) that sum to the total R 2 for the regression model and can be interpreted as relative effect sizes (LeBreton et al., 2007). Thus, we also reported the corresponding rescaled relative weights (RW J RS), which were calculated by dividing RW J by the model R 2. This latter estimate is interpreted as the proportion (or percentage when multiplied by 100) of the predicted variance (R 2 ) attributed to that variable. We also tested the statistical significance of relative weights following the confidence interval approach proposed by Tonidandel, LeBreton and Johnson (2009). All analyses were conducted using RWA- Web (Tonidandel & LeBreton, in press). Relative weights are seen as a useful supplement to traditional regression statistics (Tonidandel & LeBreton, 2011). Specifically, traditional statistics (e.g., regression weights; changes in R 2 ) can provide exaggerated or distorted estimates of variable importance when predictor variables are correlated with one another (cf. Budescu, 1993; Darlington, 1968; Johnson, 2000; Krasikova et al., 2011; LeBreton et al., 2007). In contrast, relative weights have been shown to provide more accurate and informative estimates of importance (Johnson, 2000; LeBreton, Ployhart, & Ladd, 2004). For testing H2, H4, and H6 (incremental importance), we estimated the change in R 2 obtained using hierarchical regression. To compare the incremental importance of achievementstriving over contextualized conscientiousness and vice versa, we entered the predictors into the hierarchical regression analysis using two different orders: (a) conscientiousness, contextualized conscientiousness, achievement-striving, and contextualized achievement-striving; (b) conscientiousness, achievement-striving, contextualized conscientiousness, and contextualized achievement-striving. These orders are referred to as Order 1 and Order 2. In evaluating relative importance (RW J ) and incremental validity (DR 2 ), we consider both the effect sizes and the statistical significance of these values. According to Cohen s (1992) guidelines, for multiple (partial) correlation, a value of.02 is considered small, and.15 is considered medium, which is what we use to guide our interpretation of the effect sizes for the incremental analyses (DR 2 ). Similar guidelines have not been derived for relative weights; however, based on LeBreton et al. (2007) and recommendations by scholars working in this area (S. Tonidandel, personal communication, May 14, 2014) we adopted the following heuristics: A relative weight of.01 represents a small effect, a value of.10 represents a medium effect, and a value of.30 represents a large effect. 1 For a detailed discussion of relative importance statistics and incremental importance statistics, the reader is directed to Johnson and LeBreton (2004), Krasikova, LeBreton, and Tonidandel (2011), LeBreton et al. (2007), and Tonidandel and LeBreton (2011).

5 CONTEXTUALIZATION VS. FACETS 5 Table 1. Descriptive statistics and intercorrelations. M SD GPA BARS (.79) 3. Conscientiousness **.39 ** (.81) 4. Conscientiousness context **.42 **.76 ** (.88) 5. Achievement-striving **.46 **.73 **.68 ** (.85) 6. Achievement-striving context **.45 **.65 **.82 **.75 ** (.88) Note. GPA D grade-point average; BARS D Behaviorally Anchored Rating Scale; context D items included the at school qualifier. Diagonal values indicate scale reliabilities. **p <.01. RESULTS Table 1 presents the descriptive statistics for all study variables and their intercorrelations. 2 Results for our hypothesis tests are presented in Table 2. 3 GPA as a Criterion The four predictors explained roughly 10% of the variance in GPA. The traditional importance statistics (i.e., correlations and beta weights) demonstrated that contextualized conscientiousness was slightly more important than conscientiousness, although the difference between bs was trivial. In contrast, the RWA revealed that contextualized conscientiousness was roughly two times as important as conscientiousness. Moreover, significance testing of the relative weights indicated that contextualized conscientiousness was a significant predictor, whereas conscientiousness was not. Hypothesis 1, that contextualized conscientiousness would be a relatively more important predictor than conscientiousness in predicting GPA, was supported. Consistent with Hypothesis 2, contextualized conscientiousness explained the variance of GPA above and beyond conscientiousness (DR 2 D.03, p <.01). To test Hypotheses 3 and 4, we compared relative and incremental importance between conscientiousness and achievement-striving. We found that the relative importance of achievement-striving was over twice as large as conscientiousness. Similarly, significance testing for relative weights also revealed that achievement-striving was a significant predictor, but conscientiousness was not. Achievement-striving also explained significant incremental variance (DR 2 D.03, p <.01) above and beyond conscientiousness. Thus, Hypotheses 3 and 4 were both supported. 2 There was one item included in both the general measure of conscientiousness and the facet-level measure of achievement-striving: Do just enough work to get by. Removing this overlapping item from the four measures did not result in substantive changes in the results. Detailed results of this alternative analysis are available from the authors. Given the fact that the 10-item measure of achievement striving is likely to be used by most researchers (with the overlapping item included), we only report findings based on the 10-item version here. 3 One of the reviewers asked whether the noncontextualized measures yielded incremental prediction over the contextualized measures. Our analyses showed that the average incremental validity of the global measure of conscientiousness over all possible subset models of contextualized predictors was.0008 for predicting GPA, and.0066 for predicting BARS. Moreover, comparing contextualized achievement-striving with all the remaining predictors demonstrated that the contextualized facet measure was the single most important predictor of GPA, thus, Hypothesis 5 was supported. Finally, contextualized achievement-striving showed a small but significant increment over the remaining predictors (DR 2 D.01, p <.05), supporting Hypothesis 6. Although not hypothesized, we also estimated the amount of incremental validity of contextualized achievement-striving in predicting GPA beyond (noncontextualized) achievement-striving alone (DR 2 D.03, p <.01) and beyond contextualized conscientiousness alone (DR 2 D.02, p <.01). To examine our two exploratory questions, we evaluated and compared relative and incremental importance of contextualized conscientiousness and achievement-striving. These two predictors showed largely comparable levels of predictive validities: A slightly higher correlation coefficient was found for contextualized conscientiousness as compared to achievement-striving, yet the regression coefficient and relative weights were slightly lower. BARS as a Criterion When BARS was used as the criterion, we obtained results that were consistent with those for GPA. In other words, both contextualized conscientiousness and achievement-striving showed predictive advantages over conscientiousness. With that being noted, we did observe two key differences in the results for the BARS criterion compared to the GPA criterion. First, we found mixed results for Hypotheses 5 and 6 using BARS as the criterion. Contextualized achievementstriving predicted BARS better than contextual conscientiousness, but it did not demonstrate a level of relative importance that was superior to that of achievement-striving (with no contextualization). Thus, Hypothesis 5 was not supported for BARS. On the other hand, contextualized achievement-striving still showed a small but significant increment in criterionrelated validity over the other three predictors (DR 2 D.01, p <.05), supporting Hypothesis 6. Contextualized achievement-striving incrementally predicted BARS over achievement-striving (DR 2 D.03, p <.01) and over contextualized conscientiousness (DR 2 D.04, p <.01), when no other predictors were included in the analysis. Second, when compared against each other (i.e., examining the exploratory research questions), achievement-striving predicted BARS better than contextualized conscientiousness did, as indicated in the relative weight values. Even though each of the two predictors contributed to explaining a significant amount of additional variance over the other, the amount added by achievement-striving above and beyond contextualized conscientiousness (DR 2 D.05, p <.01) was larger than the obverse (DR 2 D.02, p <.01). DISCUSSION This study extends the literature by examining whether contextual specificity might be more advantageous than content specificity. In addition, we also examined whether integrating these two approaches to specificity (i.e., contextualized facet) might further enhance the prediction of student performance outcomes. Based on our findings, we suggest that the answer

6 6 WOO, JIN, LEBRETON Table 2. Relative and incremental importance in predicting GPA and BARS. Predictors r b RW RW-RS R 2 Order1 DR 2 R 2 Order2 DR 2 Criterion: GPA Conscientiousness.21 ** % Conscientiousness context.27 ** * 24.4% ** ** Achievement-striving.26 ** * 24.9% * ** Achievement-striving context.30 **.21 *.0385 * 39.7% * * Total % Criterion: BARS Conscientiousness.39 ** * 17.2% Conscientiousness context.42 ** * 21.4% ** ** Achievement-striving.46 **.25 **.0805 * 33.0% ** ** Achievement-striving context.45 **.18 *.0693 * 28.4% * * Total % Note. GPA D grade-point average; BARS D Behaviorally Anchored Rating Scale; Order1 for hierarchical regression analysis: conscientiousness, conscientiousness context, achievement-striving, achievement-striving context. Order2 for hierarchical regression analysis: conscientiousness, achievement-striving, conscientiousness context, achievementstriving context. *p <.05. **p <.01. might depend on the specificity of the outcome. Theoretically, these results supported the well-known argument for bandwidth similarity or specificity matching that has been frequently discussed by industrial and organizational psychologists. As noted by P. C. Smith (1976), the relative degree of specificity of the predictor should be matched to the specificity of the criterion measure (p. 749), and having comparable levels of specificity between the predictor and criterion measures could enhance prediction (Hogan, Hogan, & Roberts, 1996; Ones & Viswesvaran, 1996; Schneider, Hough, & Dunnette, 1996). In our study, specificity matching is a question of whether predictive advantages of facet and contextualized measures of conscientiousness are dependent on the specificity of college student performance criterion (BARS being a broader measure than GPA, as discussed earlier). To this effect, we examined indicators of relative and incremental importance (i.e., RW and DR 2 values) first across predictors within each criterion, and then how the pattern of results might be different across the two criteria. Doing so addresses the following question: For GPA and BARS, respectively, how specific should a predictor be to achieve the maximal predictive validity for the criterion? Our data show that GPA (being a narrower or more contextualized criterion) is best predicted by the most specific personality measure included in our study (i.e., contextualized achievement-striving), whereas BARS (being a broader or less contextualized criterion) is best predicted by a noncontextualized measure of achievement-striving, which is not as specific as a contextualized measure of the same facet. This demonstrates that more specific personality measures do not always yield better predictive validity, and that predictive advantages of specificity might depend on the criterion specificity or breadth. Being conscientious and achievement-driven within the context of school most probably means putting much effort into studying, which is directly linked with student performance in the academic context or domain. However, it does not necessarily mean that the student shows high levels of performance in other contexts or domains (e.g., interpersonal and intrapersonal areas) while in college. Therefore, when the criterion measure represents a construct with high specificity, such as GPA, it would be best predicted by a construct with the comparable level of specificity (i.e., contextualized facet measure). On the other hand, for a broader behavioral criterion of college student performance, such as BARS, maximizing the specificity of predictor measures by contextualizing facets might not be ideal for enhancing its predictive validity. This study contained several important strengths. First, we tested our hypotheses using two distinct outcome measures with differing levels of specificity (i.e., GPA and BARS). In this manner, we sought to not only replicate and generalize our results, but also identify potential boundary conditions for our exploratory research questions. Due to the lack of systematic empirical research on the notion of specificity matching, we currently do not have much prior knowledge about exactly how specific the predictor has to be to match the specificity of the criterion in a given case. Thus, empirical findings on personality criterion relationships with varying degrees of specificity on both sides (e.g., those presented in this article) should continue to be documented toward a goal of building a larger database for future meta-analysis efforts (Hough & Oswald, 2008, p. 277). Second, predictors were compared based on both their relative contribution with respect to a set of other predictors (i.e., relative importance) and their unique contribution above and beyond those predictors (i.e., incremental importance). The use of relative importance analyses enabled us to directly compare correlated predictors. Such comparisons were questionable (if not taboo; Darlington, 1968) prior to the advent of RWA. These analyses provided complementary information that yielded a more complete picture of the true value of each predictor (LeBreton et al., 2007). Despite its contribution, this study is not without limitations. First, both predictor and criterion measures were collected from students self-assessments; thus the predictor criterion relationships might be potentially inflated by common method variance. The extant literature has documented that the correlation between self-reported GPA and objectively obtained GPA is quite high (average r obs D.90), and both methods of assessment tend to reveal similar patterns of correlations with other variables (Kuncel, Crede, & Thomas, 2005). Nevertheless, future research should replicate these findings with data collected from sources other than self-reports (e.g., other ratings of personality, objective measures of student

7 CONTEXTUALIZATION VS. FACETS 7 performance). Also, the fact that measures of criterion and predictor variables were administered at the same time (within the same survey administered to everyone) further limits our ability to draw causal inferences from these findings (e.g., knowing or reflecting on one s own GPA and competence as a college student might influence self-reports of conscientiousness). As such, our use of the term predictor variable should not be interpreted as suggesting or testing causal inference. In view of this limitation, we call for future research that incorporates a time-separated survey design, in which predictors are measured prior to criterion variables, or use criterion variables that do not require self-assessments. It is also possible that administering all personality predictor measures at once leads to inflated correlations among the predictors, leading to smaller incremental validities over one another. In addition, we did not control for order effects in presenting personality predictor measures (i.e., the noncontextualized measures were always presented before their contextualized measures), which might have distorted our results in some ways. Although this makes our statistical analysis a particularly conservative one (i.e., highly correlated predictors are less likely to demonstrate incremental validity over one another), future research might benefit from administering some of the personality measures in separate surveys (and possibly over multiple time points) to obtain more accurate validity estimates. A second limitation is the scope of our empirical investigation in examining the role of personality measurement specificity in predicting performance. For example, we only focused on the conscientiousness factor of the FFM. Although conscientiousness is among the most robust predictors, research has shown that other aspects of personality are also related to academic performance, such as openness (Chamorro- Premuzic & Furnham, 2008). Further, other personality traits might be particularly predictive of other specific domains of college student performance (e.g., agreeableness and extraversion predicting interpersonal outcomes; emotional stability predicting intrapersonal outcomes). As such, future research should attempt to extend these findings into other construct domains. Within the Conscientiousness domain, focusing on the achievement-striving facet also limits our contribution. Previous research suggested that other facets of Conscientiousness, such as self-discipline, are also critical for academic success (De Fruyt & Mervielde, 1996; Lievens et al., 2008; Schmit et al., 1995), and thus, our finding might or might not generalize to other facets of Conscientiousness. Future study should further clarify and document differential predictive advantages associated with different types of personality facets. Also, our selection of criterion measures was limited to college-specific contexts. Although it was beyond the scope of this study, examining criterion variables in both college and noncollege (e.g., work, family, friendship) contexts would provide a much richer understanding of how the specific context of criteria also matters in the specificity matching issue (e.g., Heller, Ferris, Brown, & Watson, 2009). Finally, our study examines whether superior predictive validity could be gained from contextualizing personality measures. However, an applied psychologist might wonder how exactly this contextualization should be done when there are multiple relevant contexts for criteria. For example, when a clinician wishes to use a contextualized personality measure to better predict various treatment outcomes, should the measure be contextualized to activities involved in the treatment itself (e.g., staying on the course of treatment; missing the appointments); to actual improvement on the symptoms that are being treated (e.g., having interpersonal issues vs. emotional issues vs. substance abuse); or to the cultural, geographic, and social environments (e.g., being in the United States vs. in Korea)? Based on the bandwidth-fidelity argument, we speculate that broader, noncontextualized measures would generally do better in predicting a wide array of outcomes. However, we also suggest that more systematic, empirical investigations will be needed to reach a definitive conclusion on such questions. In doing so, future research could utilize a similar approach to the one taken in this study where multiple types of specific personality measures are compared for relative and incremental importance. In conclusion, this study offers additional empirical evidence for the predictive advantages of two strategies of increasing specificity in personality measurement (i.e., contextualization and facets). Focusing on specific content or specific context might yield substantial gains in predictive accuracy. Whether one approach is superior to the other depends on the construct alignment between the predictor and the outcome. More important, our study shows that the combination of both strategies could (at times) yield optimal levels of predictive validity (those times being when the predictor and criterion are matched in their levels of specificity). REFERENCES Ashton, M. C. (1998). Personality and job performance: The importance of narrow traits. Summary. Journal of Organizational Behavior, 19, Bandura, A. (2001). Social cognitive theory: An agentic perspective. Annual Review of Psychology, 52, Bem, D. J., & Funder, D. C. (1978). Predicting more of the people more of the time: Assessing the personality of situations. Psychological Review, 85, Bing, M. N., Whanger, J. C., Davison, H. K., & Van Hook, J. B. (2004). Incremental validity of the frame-of-reference effect in personality scale scores: A replication and extension. Journal of Applied Psychology, 89, Budescu, D. V. (1993). Dominance analysis: A new approach to the problem of relative importance of predictors in multiple regression. Psychological Bulletin, 114, Cervone, D. (2005). Personality architecture: Within-person structures and processes. Annual Review of Psychology, 56, Chamorro-Premuzic, T., & Furnham, A. (2003). Personality traits and academic examination performance. European Journal of Personality, 17, Chamorro-Premuzic, T., & Furnham, A. (2008). Personality, intelligence and approaches to learning as predictors of academic performance. Personality and Individual Differences, 44, Cohen, J. (1992). A power primer. Psychological Bulletin, 112, doi: / Costa, P. T., & McCrae, R. R. (1992). Revised NEO Personality Inventory (NEO PI R) and NEO Five-Factor Inventory (NEO FFI) professional manual. Odessa, FL: Psychological Assessment Resources. Darlington, R. B. (1968). Multiple regression in psychological research and practice. Psychological Bulletin, 69, Davison, H. K., & Bing, M. N. (2009). Content validity does matter for the criterion-related validity of personality tests. Industrial and Organizational Psychology, 2,

8 8 WOO, JIN, LEBRETON De Fruyt, F., & Mervielde, I. (1996). Personality and interests as predictors of educational streaming and achievement. European Journal of Personality 10, Dudley, N. M., Orvis, K. A., Lebiecki, J. E., & Cortina, J. M. (2006). A metaanalytic investigation of conscientiousness in the prediction of job performance: Examining the intercorrelations and the incremental validity of narrow traits. Journal of Applied Psychology, 91, Funder, D. (2001) Personality. Annual Review of Psychology, 52, Goldberg, L. R., Johnson, J. A., Eber, H. W., Hogan, R., Ashton, M. C., Cloninger, C. R., & Gough, H. G. (2006). The international personality item pool and the future of public-domain personality measures. Journal of Research in Personality, 40, Gray, E. K., & Watson, D. (2002). General and specific traits of personality and their relation to sleep and academic performance. Journal of Personality, 70, Hampson, S. E., John, O. P., & Goldberg, L. R. (1986). Category breadth and hierarchical structure in personality: Studies of asymmetries in judgments of trait implications. Journal of Personality and Social Psychology, 51, Heggestad, E. D., & Gordon, H. L. (2008). An argument for context-specific personality assessments. Industrial and Organizational Psychology, 1, Heller, D., Ferris, D. L., Brown, D., & Watson, D. (2009). The influence of work personality on job satisfaction: Incremental validity and mediation effects. Journal of Personality, 77, Hogan, R., Hogan, J., & Roberts, B. W. (1996). Personality measurement and employment decisions: Questions and answers. American Psychologist, 51, Holtz, B. C., Ployhart, R. E., & Dominguez, A. (2005). Testing the rules of justice: The effects of frame-of-reference and pretest validity information on personality test responses and test perceptions. International Journal of Selection and Assessment, 13, Hough, L. M., & Oswald, F. L. (2008). Personality testing and industrial organizational psychology: Reflections, progress, and prospects. Industrial and Organizational Psychology, 1, Hunthausen, J. M., Truxillo, D. M., Bauer, T. N., & Hammer, L. B. (2003). A field study of frame-of-reference effects on personality test validity. Journal of Applied Psychology, 88, Johnson, J. W. (2000). A heuristic method for estimating the relative weight of predictor variables in multiple regression. Multivariate Behavioral Research, 35, Johnson, J. W., & LeBreton, J. M. (2004). History and use of relative importance indices in organizational research. Organizational Research Methods, 7, Krasikova, D., LeBreton, J. M., & Tonidandel, S. (2011). Estimating the relative importance of variables in multiple regression models. In G. P. Hodgkinson & J. K. Ford (Eds.), International review of industrial and organizational psychology (Vol. 26, pp ). Indianapolis, IN: Wiley. Kuncel, N. R., Crede, M., & Thomas, L. L. (2005). The validity of selfreported grade point averages, class ranks, and test scores: A meta-analysis and review of the literature. Review of Educational Research, 75, LeBreton, J. M., Hargis, M. B., Griepentrog, B., Oswald, F. L., & Ployhart, R. E. (2007). A multidimensional approach for evaluating variables in organizational research and practice. Personnel Psychology, 60, LeBreton, J. M., Ployhart, R. E., & Ladd, R. T. (2004). A Monte Carlo comparison of relative importance methodologies. Organizational Research Methods, 7, Lievens, F., De Corte, W., & Schollaert, E. (2008). A closer look at the frameof-reference effect in personality scale scores and validity. Journal of Applied Psychology, 93, Mischel, W. (2004). Toward an integrative science of the person. Annual Review of Psychology, 55, Mischel, W., & Shoda, Y. (1995). A cognitive affective system theory of personality: Reconceptualizing the invariances in personality and the role of situations. Psychological Review, 102, Noftle, E. E., & Robins, R. W. (2007). Personality predictors of academic outcomes: Big five correlates of GPA and SAT scores. Journal of Personality and Social Psychology, 93, O Connor, M. C., & Paunonen, S. V. (2007). Big Five personality predictors of post-secondary academic performance. Personality and Individual Differences, 43, Ones, D. S., & Viswesvaran, C. (1996). Bandwidth-fidelity dilemma in personality measurement for personnel selection. Journal of Organizational Behavior, 17, Oswald, F. L., Schmitt, N., Kim, B. H., Ramsay, L. J., & Gillespie, M. A. (2004). Developing a biodata measure and situational judgment inventory as predictors of college student performance. Journal of Applied Psychology, 89, Paunonen, S. V. (1998). Hierarchical organization of personality and prediction of behavior. Journal of Personality and Social Psychology, 74, Paunonen, S. V., & Ashton, M. C. (2001). Big five factors and facets and the prediction of behavior. Journal of Personality and Social Psychology, 81, Poropat, A. E. (2009). A meta-analysis of the five-factor model of personality and academic performance. Psychological Bulletin, 135, Roberts, B., Chernyshenko, O., Stark, S., & Goldberg, L. (2005). The structure of conscientiousness: An empirical investigation based on seven major personality questionnaires. Personnel Psychology, 58, Robie, C., Schmit, M. J., Ryan, A. M., & Zickar, M. J. (2000). Effects of item context specificity on the measurement equivalence of a personality inventory. Organizational Research Methods, 3, Schmit, M. J., Ryan, A. M., Stierwalt, S. L., & Powell, A. B. (1995). Frame of reference effects on personality scale scores and criterion-related validity. Journal of Applied Psychology, 80, Schneider, R. J., Hough, L. A., & Dunnette, M. D. (1996). Broadsided by broad traits: How to sink science in five dimensions or less. Journal of Organizational Behavior, 17, Shaffer, J. A., & Postlethwaite, B. E. (2012). A matter of context: A meta-analytic investigation of the relative validity of contextualized and noncontextualized personality measures. Personnel Psychology, 65, Smith, D. B., Hanges, P. J., & Dickson, M. W. (2001). Personnel selection and the Five-factor model: Reexamining the effects of applicant s frame of reference. Journal of Applied Psychology, 86, Smith, P. C. (1976). Behavior, results, and organizational effectiveness: The problem of criteria. In M. D. Dunnette (Ed.), Handbook of industrial and organizational psychology (pp ). Chicago, IL: Rand McNally. Tonidandel, S., & LeBreton, J. M. (2011). Relative importance analysis A useful supplement to regression analyses. Journal of Business and Psychology, 26, 1 9. Tonidandel, S., & LeBreton, J. M. (in press). RWA-Web: A free, comprehensive, web-based, and user-friendly tool for relative weight analyses. Journal of Business and Psychology. doi: /s z Tonidandel, S., LeBreton, J. M., & Johnson, J. W. (2009). Determining the statistical significance of relative weights. Psychological Methods, 14, Werner, P. D., & Pervin, L. A. (1986). The content of personality inventory items. Journal of Personality and Social Psychology, 51,