Decomposed and Holistic Job Analysis Judgments: The Moderating Role of Experience

Decomposed and Holistic Running Head: MODERATING ROLE OF WORK EXPERIENCE Decomposed and Holistic Job Analysis Judgments: The Moderating Role of Experience Matthias Spitzmuller The Eli Broad Graduate School of Management Michigan State University N475 North Business Complex East Lansing, MI 48824-22 Voice: (57) 353-6788 spitzmuller@bus.msu.edu Frederick P. Morgeson The Eli Broad Graduate School of Management Michigan State University N475 North Business Complex East Lansing, MI 48824-22 Voice: (57) 432-3520 Fax: (57) 432- morgeson@msu.edu Michael A. Campion Krannert School of Management Purdue University West Lafayette, IN 47907 Poster session presented at the 22 nd Annual Conference of the Society for Industrial and Organizational Psychology, New York, NY. April, 2007

Decomposed and Holistic 2 Poster TITLE Decomposed and Holistic Job Analysis Judgments: Experience as a Moderator ABSTRACT We investigated whether holistic judgments in job analysis are more susceptible to inflation than decomposed judgments. Moreover, we examined whether experience moderates the relationship between decomposed judgments and holistic judgments. We find that more experienced incumbents will display less convergence in their decomposed and holistic judgments than more experienced incumbents. PRESS PARAGRAPH It is a widespread belief that more experienced employees should be used in job analysis because of their superior knowledge of the job. As such, they are expected to provide more accurate judgments in job analysis. Building on decision making and memory literature, however, we suggest that the opposite might in fact be true: More experienced employees tend to use heuristics in their decision making process which can be detrimental to the overall quality of their judgment. We suggest that there is an optimal level of work experience at which job incumbents can rate their jobs with the highest accuracy.

Decomposed and Holistic 3 Job analysis is a fund amental research area in the field of industrial and organizational psychology. Also, its practical relevance for HR managers and practitioners is widely recognized (Shippmann, Ash, Battista, Carr, Eyde, Hesketh, et al., 2000). In general, job analysis describes a wide variety of systematic procedures for examining, documenting, and drawing inferences about work activities, worker attributes, and work context (Sackett & Laczo, 2003). In particular, job analysis forms the basis for such key HR processes and activities as job grading/job evaluation, hiring, performance management and compensation, the design of career succession models, or the design of individual and organizational development programs. Because of this, it is essential that job analysis information be accurate and free from bias. Accordingly, scholars have begun to investigate the range of factors that may systematically influence job analysis judgments (Morgeson & Campion, 997). At least two issues arise when investigating job analysis judgments. The first concerns what kinds of judgments are made during the data collection process. Traditionally, job analysis has focused on highly specific, decomposed judgments in which numerous specific job tasks are rated. Recently, however, there has been a trend toward more holistic judgments in which more general judgments are made. Such a trend toward holistic judgments is most evident in competency modeling efforts where very general job aspects are rated (Shippmann et al., 2000). Although research has been conducted comparing decomposed and holistic judgments, evidence in the job analysis domain has been equivocal. The second issue that arises concerns the role that job incumbent experience plays in job analysis judgments. Some have suggested that more experienced incumbents provide different information (Borman, Dorsey, & Ackerman, 992; Landy & Vasey, 99). There are two conceptional reasons for this finding: First, more experienced employees have greater knowledge

Decomposed and Holistic 4 of the job. Second, with increasing experience, job incumbents perform the job differently. Others have suggested that less experienced individuals provide more accurate information because they are more likely to systematically process information about the job (Richman & Quinones, 996). Still others have suggested that experience has no relationship with job analysis judgments (Mullins & Kimbrough, 988; Schmitt & Cohen, 989; Silverman, Wexley, & Johnson, 984). To address these gaps in the job analysis literature, we directly compare decomposed and holistic judgments using a within-subjects research design. This will add needed insight into effects of using more holistic judgments in job analysis. We then seek to understand the role of work experience in job analysis judgments by investigating how a multidimensional conceptualization and operationalization of work experience (following the work of Tesluk and Jacobs, 998) influences convergence in decomposed and holistic job analysis judgments. This contributes to the job analysis literature by enhancing our understanding of which incumbents (in terms of work experience) are best able to make holistic job analysis judgments. Human Judgment in Job Analysis Rating job content and job requirements is a complex cognitive process involving considerable rater judgment (Cornelius & Lyness, 980; Morgeson & Campion, 997). Raters must take into account numerous factors before judging the importance of various attributes of a specific job. Although it is commonly assumed that job analysis methods produce reliable and valid information, research in social and cognitive psychology has demonstrated that human judgment is limited in numerous ways. Morgeson and Campion (997) have highlighted a number of potential social and cognitive sources of inaccuracy in job analysis, and recent research suggests that some of these processes operate in job analysis contexts (Morgeson,

Decomposed and Holistic 5 Delaney-Klinger, Mayfield, Ferrara, & Campion, 2004). Consequently, connecting basic research on human judgment and decision making with job analysis research can improve our understanding of the potential sources of inaccuracy in job analysis (Cornelius & Lyness, 980). Miller (956) is among the first to acknowledge the limitations of human judgment. He suggests that humans can only handle a limited number of conceptual units when making judgments and decisions. In a similar fashion, Driver and Streufert (969) present a theoretical model according to which decision makers are no longer able to incorporate the number of external stimuli available to them when making judgments once a certain level of complexity has been reached. For the job analysis domain, this would imply that human judgment becomes increasingly unreliable as the complexity of the rating stimuli increases. Interestingly, a major recent trend in job analysis has been a movement toward competency modeling approaches (Shippmann et al., 2000). One of the key ways in which competency modeling differs from traditional methods of job analysis is in a stronger emphasis on holistic judgments. In traditional job analysis measurement, job incumbents are often asked to rate a set of very specific tasks, which are then combined into a more general behavioral category (or competency). This can be thought of as a decomposed rating strategy. In competency modeling approaches, however, job incumbents are often asked to make judgments of very general categories of behavior. This can be thought of as a holistic rating strategy. Although each approach has its strengths and weaknesses, it is clear that a holistic rating strategy is more complex and places higher cognitive demands on the rater. A long tradition of research in the decision making literature has supported the view that decomposed judgments are superior to holistic judgments (Miller, 956; Driver & Streufert, 969). Yet the superiority of decomposed judgment strategies in the domain of job analysis has

Decomposed and Holistic 6 not been fully supported. For example, some studies have found only small or non-significant differences when comparing outcomes in job analyses for decomposed or holistic judgments (Cornelius & Lyness, 980; Sackett, Cornelius, & Carron, 98). Yet other studies report significant differences between the two judgment strategies. For example, Butler and Harvey (988) found near-zero convergence between the holistic and decomposed dimension ratings (p.76). More recently, Morgeson et al. (2004) were able to show that decomposed task ratings in job analysis are less susceptible to inflation than holistic competency ratings. Given these divergent findings, additional research is needed comparing decomposed and holistic job analysis judgments for different jobs and in different organizational contexts. In addition, to investigate this issue it is important to use a within-subjects research design in which the two different types of judgments can be directly compared. We expect that decomposed job analysis judgments will be less inflated (lower mean ratings) than holistic judgments due to the reduced complexity of the judgment task. Hypothesis : Decomposed job analysis ratings will be less inflated than holistic job analysis ratings. The Role of Job Experience in Job Analysis Judgments Although we expect decomposed job analysis ratings will be less subject to inflation than holistic ratings, no previous research has investigated whether some job incumbents are better able to make holistic judgments. There is reason to believe that some incumbents are more capable of making holistic judgments than others. We propose that job experience influences the degree to which raters can make accurate holistic competency judgments. Two different arguments have been offered to explain why job experience predicts job analysis outcomes. The first, rather intuitive explanation highlights the fact that more experienced employees will know their jobs in greater detail than less experienced employees.

Decomposed and Holistic 7 Consequently, they would be in a better position to rate the content of jobs in job analysis. Also, Landy and Vasey (99), Borman, Dorsey, and Ackerman (992) note that with increasing experience, incumbents tend to emphasize different aspects of their jobs. An alternative explanation has been offered more recently by Richman and Quinones (996). Drawing on research into human memory, they suggest that increased experience leads to routines that make it very difficult for job incumbents to remember the frequency with which certain tasks have been performed. In a lab experiment, Richman and Quinones found support for their hypothesis that low experience participants would evidence greater accuracy than participants in the high experience condition. These findings are consistent with research into decision making heuristics (Tversky & Kahnemann, 974). In short, decision-makers often rely on heuristics which mirror the decision situation only imperfectly. Morgeson and Campion (997) suggested that more experienced workers would tend to use heuristics more frequently when rating jobs than less experienced workers. Consequently, the accuracy of job analysis outcomes will be affected in a number of ways. Job information may be incomplete, there may not be adequate discriminate among jobs, and job analysis judgments may become inaccurate. These propositions, however, have not yet been subject to empirical test. It is likely that the frequent use of (experienced) subject matter experts in job analysis (Maurer & Tross, 2000; Shippmann et al., 2000) and the dominant belief that more experienced employees will provide more accurate judgments of jobs and job requirements has contributed to the lack of empirical testing. If one were to investigate work experience, however, an important question concerns how to conceptualize and operationalize job experience. Past research has relied almost exclusively on unidimensional measures of job experience, such as tenure in the organization. Tesluk and

Decomposed and Holistic 8 Jacobs (998, p. 323), however, emphasize the utility of conceptualizing experience in complex, multidimensional terms, arguing that different types of experience measures capture different types of work-relevant experience. Also, Tesluk and Jacobs advocate the use of experience measures which reflect the same level of specification as the outcome criterion. They specify different levels of job experience, such as experience on the task, on the job, in the work group, or in the organization. Tesluk and Jacobs suggest that it is important to match the level of specification in the experience measure with the level of specification in the criterion. Based on their work, we feel it is important to broaden the operationalization of experience measures in the job analysis domain and use multiple measures of experience that reflect experience on the task and on the job. We hypothesize that these experience measures will moderate the relationship between decomposed and holistic job analysis ratings. Hypothesis 2: Task experience will moderate the relationship between decomposed and holistic job analysis ratings such that incumbents with more task experience will display less convergence in their decomposed and holistic judgments than incumbents with less task experience. Hypothesis 3: Job tenure will moderate the relationship between decomposed and holistic job analysis ratings such that incumbents with more job tenure will display less convergence in their decomposed and holistic judgments than incumbents with less job tenure. Hypothesis 4: Previous job experience will moderate the relationship between decomposed and holistic job analysis ratings such that incumbents with more previous job experience will display less convergence in their decomposed and holistic judgments than incumbents who have less previous job experience. Overall organizational or occupational experience, on the other hand, will be less useful in understanding the convergence between decomposed and holistic judgments because they do not match the level of specification of the criterion. In other words, experience at the organizational or occupational level is likely to have little impact on judgments about the job.

Decomposed and Holistic 9 We hypothesize that these experience measures will not moderate the differential relationship between decomposed task ratings and holistic competency judgments. Participants Hypothesis 5: Specific organizational experience, total organizational experience, or career experience, will not moderate the relationship between decomposed and holistic job analysis ratings. METHOD Job analysis surveys were distributed to 88 employees of a governmental agency involved in international economic development. A total of 33 employees responded to the survey, yielding a response rate of 7%. Some employees did not complete the entire survey, so the final sample size for respondents with complete data on all measures was 2. The average respondent has performed tasks similar to the current job for over 9 years. The average tenure in the organization is over years and the average tenure in the current job is over 2 years. Respondents average 5 years previous experience and over 2 years career experience. The average person has had more than 3 similar jobs and has worked in nearly 5 different organizations. Demographically, the sample is 76% men and 87% White. Measures An existing job analysis survey was updated based on the input of subject matter expert committee meetings. The survey contained eight components that comprised the major job duties (Table ). For each component, a list of individual task items was included to decompose the larger component into smaller elements. Four to twenty specific items were included to decompose component into its essential tasks. The average of the specific item ratings for each component constitutes the decomposed ratings.

Decomposed and Holistic 0 The overall components were then rated at the end of the survey. The component label plus a brief summary of the individual items was listed. The rating of the component constitutes the holistic rating. Thus, the averaged component items and the holistic rating of the component referenced identical information, just at different levels of specificity (i.e., decomposed vs. holistic). The eight job components were rated in terms of their importance to the job. A 5-point importance scale was used for both decomposed and holistic ratings, where 5 = extremely important, 4 = very important, 3 = important, 2 = somewhat important, = not important. Average internal consistency reliability for the decomposed ratings across the eight job components ranged from.89 to.94. At the end of the job analysis survey respondents indicated the extent of their work experience. As indicated earlier, task and job experience were operationalized in three ways. Task experience was measured with the following question: How long have you been performing tasks like the tasks in your current job? Job tenure was measured with the following question How long have you worked at your current job? Previous job experience was measured with the following question: How many different jobs like your current job have you had? Organizational and occupational experience was operationalized in three ways. Organizational experience was measured with the following question: How long have you worked at the [organization name]? Total organizational experience was measured with the following question: How many different organizations have you worked for (in total)? Career experience was measured with the following question: How long have you been working in your entire career (including your current job and all previous jobs)?

Decomposed and Holistic RESULTS Table 2 contains descriptives and correlations for all study measures. Hypothesis predicted that ratings of holistic components are more likely to be inflated than decomposed task items. We tested this hypothesis by examining the differences between the mean of the decomposed ratings and the holistic rating. We used a paired samples t-test to analyze the mean differences between holistic component ratings and decomposed task ratings. The results are summarized in Table 3. Hypothesis was supported for all eight job components (p <.0). The average mean difference between the scales including decomposed ratings and the respective holistic rating was.52 (on a five-point scale), indicating a substantial inflation of the holistic ratings. Hypotheses 2, 3, and 4 predicted that three experience measures (task experience, job tenure, previous job experience) would moderate the relationship between decomposed and holistic ratings. We tested these hypotheses by conducting a series of moderation regression analyses in which we entered the main effects in step one, followed by the interaction term in step two. These results are summarized in Table 4. All three hypotheses are supported: For virtually all components (seven out of eight), job experience measures moderate the relationship between decomposed and holistic ratings such that more experienced employees converge less in their ratings of decomposed items and holistic components. An example for this interaction effect is illustrated in figure. The individual experience measures are able to explain a significant amount of additional variance (up to 5.4% for one individual experience measure). Considering the size of interaction effects tyically found in our field, we can conclude that the experience measures have a

Decomposed and Holistic 2 substantial impact on the relationship between decomposed task items and holistic component ratings. While not all experience measures moderate all possible relationships, at least one of the three does so for each component (with the exception of the component analysis we will discuss this finding later). For two components (office/staff management and customer service/support), all three experience measures moderate the relationship between decomposed and holistic ratings. In a similar fashion, we tested Hypothesis 5, in which we predicted that the experience measures reflecting tenure in the organization, the workgroup or experience in the profession do not moderate this relationship as they refer to different levels of specification as described by Tesluk & Jacobs (998). Results for this analysis are summarized in Table 4. As predicted, none of these experience measures moderated the decomposed and holistic ratings for any of the job components, providing strong support for Hypothesis 5. This provides the first empirical support for Tesluk and Jacobs (998) suggestion that the level of specificity in measures of experience is important to consider. While we did not hypothesize this initially, we examined post hoc for which components the moderating effect of the three experience measures was particularly emphasized and for which component(s) the effect was rather weak. None of the three experience measures moderates the relationship between decomposed and holistic job analysis ratings for the component analysis. Also, effect sizes for the moderating effect of experience are rather small for the components interagency cooperation and policy development (only one experience measure moderates the hypothesized relationship, p <., and advocacy dispute resolution and diplomacy (only one experience measure moderates the hypothesized relationship, p <.05. On the other hand, the moderating effect of the experience variables was strong for the components

Decomposed and Holistic 3 office staff/management, budget/fiscal, customer service/support, and program development. Analyzing the individual items which form the overall components suggests the following: The more complex one component, the stronger the hypothesized moderating effect of experience measures. The component analysis includes rather simple items, such as conducts pre-licensing and post-shipment checks, analyzes infrastructure development to assess opportunities for US businesses. On the other hand, the component customer service and support includes more complex items, such as considers customer feedback for purposes of individual professional development and improvement, or identifies customer expectations. As discussed above, decision makers use heuristics especially when dealing with higher levels of complexity. As we were able to show, experienced employees use heuristics more often than less experienced employees. As heuristics become more salient in complex decisions, experience measures seem to moderate the hypothesized relationship between decomposed and holistic judgment particularly well for complex task components.relationships). DISCUSSION Our study adds to the job analysis literature in a number of ways. First, we provide further evidence that holistic judgments are inflated relative to decomposed judgments. Our findings suggest that techniques that rely on more holistic judgments (i.e., competency modeling) may result in inflated work requirements. Second, we find that people with less experience displayed more convergence in their decomposed and holistic judgments. This finding is theoretically meaningful because it suggests that individuals with higher levels of work experience may be overly relying on simplifying heuristics when making holistic judgments. Moreover, our findings offer support for some of the

Decomposed and Holistic 4 cognitive limitations outlined in Morgeson and Campion (997), specifically the cognitive limiations of decision makers in job analysis. Third, we showed the usefulness of a multidimensional conceptualization of work experience. A multidimensional conceptualization of work is useful for three reasons. First, only relying on one measure of work experience to investigate these issues would portray an incomplete picture of the moderating effect of work experience. Second, this is the first study to include the more complex conceptualization of work experience noted by Tesluk and Jacobs (998). Third, the fact that we found no moderating effect for organizational or occupational experience provides the first empirical support for Tesluk and Jacobs (998) suggestion that the level of specificity in measures of experience is important to consider. Fourth, our findings have important theoretical implications for the study of decomposed and holistic judgment in that they improve our understanding of the relationship between decomposed and holistic judgments. Moreover, our findings helps reconcile past research on experience in the job analysis context, thereby extending job analysis theory. Finally, we believe that our article has important practical implications. Our findings provide strong support for the notion that more experience may not always be better for raters in job analysis. Instead, we suggest that there may be an optimal level of job experience at which raters can describe job content and job requirements with the highest accuracy. We hope that our study serves as an impetus for future research in this area.

Decomposed and Holistic 5 References Borman, W. C., Dorsey, D., & Ackerman, L. (992). Time-spent responses as time allocation strategies: Relations with sales performance in a stockbroker sample. Personnel Psychology, 45, 763-777. Butler, S. K., & Harvey, R. J. (988). A comparison of holistic versus decomposed rating of Position Analysis Questionnaire work dimensions. Personnel Psychology, 4, 76-77. Cornelius, E. T., & Lyness, K. S. (980). A comparison of holistic and decomposed judgment strategies in job analyses by job incumbents. Journal of Applied Psychology, 65, 55-63. Driver, M. J., & Streufert, S. (969). Integrative complexity: An approach to individuals and groups as information processing systems. Administrative Science Quarterly, 4, 272-285. Landy, F. J., & Vasey, J. (99). Job Analysis: The composition of SME samples. Personnel Psychology, 44, 27-50. Maurer, T. J., & Tross, S. A. (2000). The relationship between SME job experience and job analysis ratings: Findings with and without statistical control. Journal of Business and Psychology, 5, 97-0. Miller, G. A. (956). The magical number of seven, plus or minus two: Some limits on our capacity for processing information. The Psychological Review, 63, 8-97. Morgeson, F. P., & Campion, M. A. (997). Social and cognitive sources of potential inaccuracy in job analysis. Journal of Applied Psychology, 82, 627-655.

Decomposed and Holistic 6 Morgeson, F. P., Delaney-Klinger, K., Mayfield, M. S., Ferrara, Ph., & Campion, M. A. (2004). Self-presentation in job analysis: A field experiment investigating inflation in abilities, tasks, and competencies. Journal of Applied Psychology, 89, 674-686. Mullins, W. C., & Kimbrough, W. W. (988). Group composition as a determinant of job analysis outcomes. Journal of Applied Psychology, 73, 657-664. Richman, W. L., & Quinones, M. A. (996). Task frequency rating accuracy: The effect of task engagement and task experience. Journal of Applied Psychology, 8, 52-524. Sackett, P. R., Cornelius, E. T., & Carron, T. J. (98). A comparison of global judgment vs. task oriented approaches to job classification. Personnel Psychology, 34, 79-804. Sackett, P.R., & Laczo, R. M. (2003). Job and work analysis. In W. C. Borman, D. R. Ilgen, R. J. Klimoski (Eds.), Handbook of psychology, volume 2, industrial and organizational psychology, 2-37. Hoboken, NJ: Wiley. Schmitt, N., & Cohen, S. A. (989). Internal analysis of task ratings by job incumbents. Journal of Applied Psychology, 74, 96-04. Shippmann, J. S., Ash, R. A., Battista, M., Carr, L., Eyde, L. D., Hesketh, B., et al. (2000). The practice of competency modelling. Personnel Psychology, 53, 703-740. Silverman, S. B., Wexley, K. N., & Johnson, J. C. (984). The effects of age and job experience on employee responses to a structured job analysis questionnaire. Public Personnel Management, 3, 355-359. Tesluk, P. E., & Jacobs, R. R. (998). Toward an integrated model of work experience. Personnel Psychology, 5, 32-355. Tversky, A., & Kahneman, D. (974). Judgment under uncertainty: Heuristics and biases. Science, 85, 24-3.

Decomposed and Holistic 7 Table Major Job Components. Office and Staff Management 2. Budget and Fiscal 3. Program Development 4. Interagency Cooperation and Policy Development 5. Developing Partnerships and Contacts 6. Advocacy and Dispute Resolution and Diplomacy 7. Analysis 8. Customer Service and Support

Decomposed and Holistic 8 Table 2 Mean, standard deviation and intercorrelations among study variables 2 3 4 5 6 7 8 9 0 2 3 4 5 6 7 8 9 20 2 22 office staff management - decomposed items.00 budget fiscal - 2 decomposed items 0.76**.00 program development - 3 decomposed items 0.68** 0.46**.00 analysis - 4 decomposed 0.6** 0.43** 0.86**.00 interagency cooperation and policy development - 5 decomposed items 0.64** 0.56** 0.63** 0.7**.00 customer service and support - 6 decomposed items 0.59** 0.40** 0.78** 0.70** 0.52**.00 developing partnerships and contacts - 7 decomposed items 0.59** 0.38** 0.85** 0.85** 0.7** 0.7**.00 advocacy dispute resolution diplomacy - 8 decomposed items 0.55** 0.46** 0.67** 0.75** 0.82** 0.52** 0.80**.00 9 osm - holistic 0.4** 0.33** 0.42** 0.4** 0.28** 0.5** 0.4** 0.25**.00 0 bf - holistic 0.50** 0.60** 0.37** 0.39** 0.35** 0.4** 0.35** 0.29** 0.66**.00 pd - holistic 0.56** 0.4** 0.58** 0.54** 0.40** 0.54** 0.54** 0.38** 0.68** 0.67**.00 2 an - holistic 0.43** 0.4** 0.45** 0.45** 0.42** 0.43** 0.50** 0.42** 0.53** 0.58** 0.57**.00 3 icpd - holistic 0.44** 0.4** 0.38** 0.38** 0.47** 0.43** 0.45** 0.44** 0.62** 0.60** 0.54** 0.64**.00 4 css - holistic 0.43** 0.29** 0.44** 0.37** 0.30** 0.58** 0.4** 0.25** 0.47** 0.39** 0.6** 0.59** 0.46**.00 5 dpc - holistic 0.50** 0.27** 0.54** 0.48** 0.40** 0.53** 0.60** 0.43** 0.55** 0.46** 0.63** 0.54** 0.46** 0.54**.00 6 adrd - holistic 0.38** 0.38** 0.45** 0.45** 0.50** 0.55** 0.46** 0.55** 0.56** 0.56** 0.52** 0.57** 0.6** 0.44** 0.6**.00 7 Different Jobs 0.22* 0.22* 0.7* 0.2* 0.8* 0.29** 0.7** 0.2 0.2* 0.2 0.8** 0.08 0.08 0.0 0.6 0.4.00 8 Different Orgs -0. -0.04-0.3-0.04-0. -0.07 0.02-0.07-0.08-0.3-0.25** -0.02-0.3-0.03-0.08-0.3 0.07.00 9 Years Doing Tasks 0.28** 0.24** 0.6 0.4 0.4 0.22* 0.8* 0.6 0.6 0.6 0.9* 0.09 0.03 0.3 0.4 0.0 0.38** 0.00.00 20 Years In Job 0.07-0.07 0.07 0.04-0.03 0.05 0.05 0.05 0.4 0.3 0.8* 0.6 0.7 0.2* 0.05 0.09-0.08-0.07 0.8*.00 2 Years at FCS 0.26** 0.29** 0.05 0.3 0.20** 0.0 0.07 0.6 0.06 0.23* 0.2* 0.0 0.08 0.08 0.60 0.06 0.20* -0.22* 0.54** 0.2.00 22 Years in Previous Job 0.09 0.07 0.08 0. 0.6 0. 0.07 0.5 0.03 0.07 0.20* 0.06 0. 0.2* 0.00 0.08 0.0-0.23* 0.9* 0.27** 0.09.00 23 Years in Career 0. 0.6-0.06 0.02 0.0 0.05 0.03 0.07 0.05 0.20* 0.2 0.0 0.20* 0.2* 0.07 0.07 0.23* 0.0 0.49** 0. 0.59** 0.33** Mean 3.90 3.40 3.96 3.82 3.63 4.38 4.09 3.66 4.68 4.2 4.7 4.20 4.22 4.54 4.52 4.40 3.6 4.64 9.24 2.09.8 5.6 2 SD 0.58 0.96 0.70 0.82 0.87 0.73 0.77 0.98 0.64 0.97 0.86 0.96 0.93 0.73 0.70 0.79 2.0 2.05 7.58 2.66 6.09 4.0

Decomposed and Holistic 9 Table 3 Paired samples t-test comparing holistic and decomposed judgment Paired Samples Test Pair Pair 2 Pair 3 Pair 4 Pair 5 Pair 6 Pair 7 Pair 8 Paired Differences 95% Confidence Interval Mean SD Std. Error Mean of the Difference t df Sig. (2-tailed) Lower Upper office staff management: holistic vs. decomposed 0.78 0.74 0.08 0.63 0.93 0.30 94.00 0.00 budget fiscal: holistic vs. decomposed 0.8 0.90 0.09 0.63.00 8.78 94.00 0.00 program development holistic: holistic vs. decomposed 0.2 0.75 0.08 0.06 0.36 2.74 94.00 0.0 interagency cooperation and policy development: holistic vs. decomposed 0.59 0.98 0.0 0.39 0.79 5.89 94.00 0.00 developing partnerships and contacts: holistic vs. decomposed 0.42 0.74 0.08 0.27 0.57 5.6 94.00 0.00 advocacy dispute resolution diplomacy: holistic vs. decomposed 0.75.0 0.0 0.54 0.95 7.2 93.00 0.00 analysis importance: holistic vs. decomposed 0.38 0.97 0.0 0.8 0.58 3.80 94.00 0.00 customer service and support: holistic vs. decomposed 0.6 0.7 0.07 0.0 0.30 2.3 94.00 0.04

Decomposed and Holistic 20 Table 4 Regression results regressing holistic rating components on decomposed ratings, experience measures and the interaction (Hypotheses 2, 3, 4) Regression - Dependent Variable: Office Staff Management Holistic Step Variable B Change Step Variable B Change Step Variable B Change office staff management - decomposed 0.694** office staff management - decomposed 0.669** office staff management - decomposed 0.907** Years Doing Tasks 0.09* 0.22 Years in Job 0.407** 0.232 Different Jobs 0.748** 0.232 2 osm_x_tskexp -0.026* 0.023 2 osm_x_jobexp -0.092** 0.038 2 osm_x_diffjob -0.77** 0.40 Regression 2 - Dependent Variable: Budget Fiscal Holistic Step Variable B Change Step Variable B Change Step Variable B Change budget fiscal - decomposed 0.822** budget fiscal - decomposed 0.834** budget fiscal - decomposed 0.755** Years Doing Tasks 0.094** 0.382 Years in Job 0.306** 0.44 Different Jobs 0.52 0.382 2 bf_x_tskexp -0.025** 0.025 2 bf_x_jobexp -0.072** 0.054 2 bf_x_diffjob -0.045 0.009 Regression 3 - Dependent Variable: Program Development Holistic Step Variable B Change Step Variable B Change Step Variable B Change program development - decomposed 0.684** program development - decomposed 0.905** program development - decomposed 0.935** Years Doing Tasks 0.02 0.354 Years in Job 0.578** 0.365 Different Jobs 0.523** 0.352 2 pd_x_tskexp 0.000 0.000 2 pd_x_jobexp -0.23** 0.027 2 pd_x_diffjob -0.8** 0.046 Regression 4 - Dependent Variable: Analysis Holistic Step Variable B Change Step Variable B Change Step Variable B Change Analysis - decomposed 0.365** Analysis - decomposed 0.667** Analysis - decomposed 0.646** Years Doing Tasks -0.070 0.96 Years in Job 0.464* 0.26 Different Jobs 0.274 0.95 2 an_x_tskexp 0.09 0.02 2 an_x_jobexp -0.02 0.08 2 an_x_diffjob -0.068 0.06

Decomposed and Holistic 2 Table 4 (continued): Regression results regressing holistic rating components on decomposed ratings, experience measures and the interaction (Hypotheses 2, 3, 4) Regression 5 - Dependent Variable: Interagency Cooperation and Policy Development Holistic Step Variable B Change Step Variable B Change Step Variable B Change ICPD - decomposed 0.334** ICPD - decomposed 0.577** ICPD - decomposed 0.589** Years Doing Tasks -0.079* 0.24 Years in Job 0.25 0.245 Different Jobs 0.55 0.23 2 icpd_x_tskexp 0.02* 0.02 2 icpd_x_jobexp -0.052 0.00 2 icpd_x_diffjob -0.042 0.008 Regression 6 - Dependent Variable: Customer Service and Support Holistic Step Variable B Change Step Variable B Change Step Variable B Change Customer Service and Customer Service and Customer Service and Support - Support - Support - decomposed 0.433** decomposed 0.78** decomposed 0.776** Years Doing Tasks -0.22** 0.365 Years in Job 0.626** 0.393 Different Jobs 0.462** 0.365 2 css_x_tskexp 0.027** 0.028 2 css_x_jobexp -0.28** 0.043 2 css_x_diffjob -0.00** 0.049 Regression 7 - Dependent Variable: Developing Partnerships and Contacts Holistic Step Variable B Change Step Variable B Change Step Variable B Change Developing Partnerships Developing Partnerships Developing Partnerships and Contacts - and Contacts - and Contacts - decomposed 0.63** decomposed 0.662** decomposed 0.83** Years Doing Tasks 0.027 0.355 Years in Job 0.202 0.354 Different Jobs 0.46** 0.358 2 dpc_x_tskexp -0.006 0.00 2 dpc_x_jobexp -0.046 0.007 2 dpc_x_diffjob -0.03** 0.05 Regression 8 - Dependent Variable: Advocacy Dispute Resolution and Diplomacy Holistic Step Variable B Change Step Variable B Change Step Variable B Change ADRD - decomposed 0.326** ADRD - decomposed 0.379** ADRD - decomposed 0.625** Years Doing Tasks -0.05 0.26 Years in Job -0.030 0.26 Different Jobs 0.372** 0.22 2 adrd_x_tskexp 0.0 0.009 2 adrd_x_jobexp 0.04 0.00 2 adrd_x_diffjob -0.087** 0.046

Decomposed and Holistic 22 Table 5: Regression results regressing holistic rating components on decomposed ratings, experience measures and the interaction (Hypotheses 5) Regression - Dependent Variable: Office Staff Management Holistic Step Variable B Change Step Variable B Change Step Variable B Change office staff management - office staff management - office staff management - decomposed 0.88** decomposed 0.355 decomposed 0.697** Different Organizations 0.228 0.22 Years in Organization -0.077 0.224 Years in career 0.030 0.220 2 osm_x_difforg -0.063 0.009 2 osm_x_t_org 0.08 0.006 2 osm_x_t_career -0.008 0.003 Regression 2 - Dependent Variable: Budget Fiscal Holistic Step Variable B Change Step Variable B Change Step Variable B Change budget fiscal - budget fiscal - office staff management - decomposed 0.353* decomposed 0.793** 0.385 decomposed 0.783** Different Organizations -0.245* 0.392 Years in Organization 0.069 0.007 Years in career 0.04 0.393 2 bf_x_difforg 0.060 0.03 2 bf_x_t_org -0.08 2 osm_x_t_career -0.008 0.003 Regression 3 - Dependent Variable: Program Development Holistic Step Variable B Change Step Variable B Change Step Variable B Change program development - program development - program development - decomposed.06** decomposed 0.562** decomposed 0.924** Different Organizations 0.93 0.376 Years in Organization -0.09 0.378 Years in career 0.06 0.37 2 pd_x_difforg -0.070 0.04 2 pd_x_t_org 0.02 0.003 2 osm_x_t_career -0.00 0.004 Regression 4 - Dependent Variable: Analysis Holistic Step Variable B Change Step Variable B Change Step Variable B Change Analysis - Analysis - Analysis - decomposed 0.853** decomposed 0.395* decomposed 0.70** Different Organizations 0.275 0.95 Years in Organization -0.047 0.97 Years in career 0.055 0.204 2 an_x_difforg -0.073 0.06 2 an_x_t_org 0.0 0.003 2 osm_x_t_career -0.0 0.006

Decomposed and Holistic 23 Table 5 (continued): Regression results regressing holistic rating components on decomposed ratings, experience measures and the interaction (Hypotheses 5) Regression 5 - Dependent Variable: Interagency Cooperation and Policy Development Holistic Step Variable B Change Step Variable B Change Step Variable B Change ICPD - ICPD - ICPD - decomposed 0.53** decomposed 0.304* decomposed 0.299 Different Organizations 0.006 0.29 Years in Organization -0.070 0.23 Years in career -0.02 0.235 2 icpd_x_difforg -0.02 0.000 2 icpd_x_t_org 0.08 0.00 2 osm_x_t_career 0.009 0.005 Regression 6 - Dependent Variable: Customer Service and Support Holistic Step Variable B Change Step Variable B Change Step Variable B Change Customer Service and Customer Service and Customer Service and Support - Support - Support - decomposed 0.784** decomposed 0.502** decomposed 0.679** Different Organizations 0.66 0.365 Years in Organization -0.035 0.365 Years in career 0.039 0.394 2 css_x_difforg -0.037 0.005 2 css_x_t_org 0.009 0.002 2 osm_x_t_career -0.005 0.00 Regression 7 - Dependent Variable: Developing Partnerships and Contacts Holistic Step Variable B Change Step Variable B Change Step Variable B Change Developing Partnerships and Contacts - Developing Partnerships and Contacts - Developing Partnerships and Contacts - decomposed 0.566** decomposed 0.492** decomposed 0.387* Different Organizations -0.063 0.366 Years in Organization -0.032 0.354 Years in career -0.035 0.355 2 dpc_x_difforg 0.005 0.000 2 dpc_x_t_org 0.009 0.002 2 osm_x_t_career 0.00 0.006 Regression 8 - Dependent Variable: Advocacy Dispute Resolution and Diplomacy Holistic Step Variable B Change Step Variable B Change Step Variable B Change ADRD - ADRD - ADRD - decomposed 0.234 decomposed 0.294** decomposed 0.495** Different Organizations -0.65 0.22 Years in Organization -0.043 0.22 Years in career 0.02 0.23 2 adrd_x_difforg 0.034 0.009 2 adrd_x_t_org 0.0 0.006 2 osm_x_t_career -0.005 0.002

Decomposed and Holistic 24 Figure Example of the moderating effect of job experience on the relationship between decomposed task ratings and holistic ratings of components. 5.2 5 4.8 Office staff management - holistic 4.6 4.4 4.2 4 Job Tenure Low High 3.8 3.6 Office staff management - decomposed Low High