Enhancing Cost Estimation Models with Task Assignment Information

Transcription

1 Enhancing Cost Estimation Models with Task Assignment Information Joanne Hale Area of MIS Culverhouse College of Commerce and Business Administration The University of Alabama Tuscaloosa, AL Voice: (205) Fax: (205) Allen Parrish Brandon Dixon Department of Computer Science The University of Alabama Tuscaloosa, AL Voice: (205) Fax: (205) Randy K. Smith Mathematical, Computing and Information Sciences Department 301 Bibb Graves Hall Jacksonville State University 700 Pelham Road North Jacksonville, AL Voice: (256) June 2000

2 Enhancing Cost Estimation Models with Task Assignment Information Joanne Hale Area of MIS Culverhouse College of Commerce and Business Administration The University of Alabama Tuscaloosa, AL Voice (205) Fax (205) Allen Parrish Brandon Dixon Department of Computer Science The University of Alabama PO Box Tuscaloosa, AL Voice (205) Fax (205) Randy K. Smith Mathematical, Computing and Information Sciences Department 301 Bibb Graves Hall Jacksonville State University 700 Pelham Road North Jacksonville, AL Voice (256) June 2000 Abstract Team task assignments appear to have a significant potential impact on the overall success of a project. For example, it is frequently necessary for an individual to work concurrently on a number of different tasks. Conversely, it is also frequently necessary for several individuals to work together on the same task. In previous work, we examined the relationship between task assignments and overall project effort. In this paper, we propose adjustment factors based on task assignment patterns that can be used to tune existing cost estimation models. We show significant improvements in the predictive abilities of both COCOMO I and II by enhancing them with these factors. Keywords: Software effort estimation, COCOMO, project task assignment. It is a well-accepted project management axiom that increasing the number of people working concurrently on a task does not result in a corresponding increase in productivity (e.g., Brooks Law: Adding more programmers to a late project makes it later [4]). In a similar vein, there is an expectation that having individuals focused on a single task is an important mechanism to foster productivity [6]. We refer to the manner in which tasks are shared among team members as a task assignment pattern. Although certain task assignment patterns are widely cited as desirable in the software engineering literature, there is remarkably little in the way of supporting empirical studies. In previous work, we reported on an empirical study in this area, identifying three new metrics to measure various dimensions of a given task assignment pattern [9]. We called the three metrics in [9] intensity, concurrency and fragmentation. Intensity measures the degree of focus on a particular task. A task with a high level of intensity is worked on continuously, while tasks with low levels of intensity have frequent breaks in the work activity (perhaps to allow work to be completed on other tasks). Concurrency refers to the degree to which team members are working together cooperatively on given tasks. Fragmentation measures the degree to which a team s time is fragmented across multiple tasks. In [9], we discovered that these metrics substantially impact development effort, significantly improving the performance of the Intermediate COCOMO I estimation model [2,3]. 1

3 Derived from our previous work [9], we now propose a simple adjustment process for scaling the results of existing effort estimation models to reflect the impact of our task assignment metrics. Because it is a simple multiplicative scaling process, its application to the results of existing estimation models is straightforward. Consequently, existing models, which do not take into account task assignment patterns, can be adjusted to reflect what has already been found to impact development effort. The body of the paper is organized as follows. The next section introduces our three metrics: intensity, concurrency and fragmentation, and discusses their impact on effort estimation as presented in [9]. The following section then introduces the corresponding effort adjustment factors, while conclusions are presented in the final section. TASK ASSIGNMENT METRICS In a previously reported empirical study [9], we found three task assignment factors: concurrency, intensity and fragmentation, which significantly improved both the overall fit and predictive ability of the Intermediate COCOMO I effort estimation model. These factors were measured with respect to a group of developers, with the results of the measurements used to augment existing effort estimation models. We assume that a task is some separately identifiable activity, which could include a software module or application, or a particular development activity. While the characterization of an individual s activities into discrete, independent tasks may be difficult in some cases, we require that the developer construct such a characterization in order for our factors to be quantitatively measured. In [9], the group of developers was employed by a contractor where billing logs were kept; we used such billing logs as a basis for computing the task assignment metrics. By augmenting an Intermediate COCOMO I model with task assignment metrics (TAMs) calibrated to the specific development environment: the Average Square Root Error [7] was reduced from 8.5 to 6.1 1, and the percentage of task estimates having a Magnitude of Relative Error [8] of 30% or less was improved from 31% to 49% 2. Model 1: Intermediate COCOMO 1 Model 2: Calibrated COCOMO I Goodness of Fit Model 3: Calibrated COCOMO (plus 3 TAMs) Average Square Root Error Quality of Estimation PRED(30%) 31% 36% 49% Table 1. Model Comparison 1 The Average Square Root Error (ASRE) measures the ability of the model to fit the data from which it was derived. In a perfect prediction, the ASRE approaches 1. 2 The Magnitude of Relative Error is the size of the deviation between the predicted effort and the actual effort for a particular task t. Predicted effort is calculated with task t removed from the data set. The notation PRED(X%) refers to the percentage of task estimates having a deviation less than X%. 2

4 Formal definitions of our three task assignment metrics are given in [9]. Here we provide the basic substance of those definitions. A key underlying concept in all three definitions is the notion of a time unit, which is a discrete interval of time of some fixed duration. A task is characterized by its duration, the span between the time unit when work on a task is begun and the time unit when the task is completed. Our task assignment metrics are defined with respect to tasks (i.e., they are properties of tasks, rather than of individuals). This allows them to serve as dependent variables in task effort prediction models. Intensity (for task t) is then defined as: Number of time units devoted to t Total duration of t A task with a high intensity level is worked on with sharp focus and few or no hiatuses, while a low intensity level is associated with a task that may have sat untouched for long periods of time. In our study, we found that development effort decreased as intensity increased. This demonstrated impact of intensity supports the work of DeMarco and Lister [6], who in their survey of development projects, discovered that uninterrupted time dramatically increases productivity. High productivity requires single-minded work time; thus, every interruption costs additional immersion time before the programmer can again single-mindedly focus on the task at hand. Concurrency (for task t at time unit x) is defined as the number of developers working on task t during time unit x. To obtain the concurrency for task t over its entire duration, the concurrency for all time units when work was reported on t is averaged. While concurrency does not directly measure interactions among team members, a strong relationship is expected between interactions and the degree to which team members report simultaneous activity on the task. We found that within the environment studied, increased concurrency (reflecting a higher degree of team collaboration) resulted in greater development effort. Thus, teams took less time to complete a task when members were able to work more independently. This may be due to a lack of team discipline, as suggested by the work of Basili and Reiter [1], or to programmers natural tendency to prefer working alone, as suggested by Cougar and Adelsberger [5]. However, the impact on effort estimation is clear: greater concurrency leads to greater development effort. The fragmentation metric measures the degree to which a team s time is fragmented over multiple tasks. To determine the fragmentation associated with task t, we determine the number of tasks reported by each developer during each time unit when work is reported on t. The sum of all such reported tasks is computed for each developer (SumDev i is the sum of the tasks reported for the i th developer). A team average is then computed to determine the fragmentation for task t. Assuming n developers, this average is computed as follows: 3

5 n i =1 SumDev n In our study, development effort was found to increase with fragmentation. As a developer s time is divided over greater numbers of unrelated activities, less time can be spent on productive work because more time must be spent shifting between tasks. The sidebar 3 contains an example showing how to compute values for intensity, concurrency and fragmentation. Table 2 below summarizes the three task assignment metrics. i Task Assignment Description Impact on Effort Factor Intensity Degree of schedule compression Development effort decreases with intensity Concurrency Degree of coordination between team members Development effort increases with concurrency Fragmentation Degree to which time is fragmented across multiple tasks Development effort increases with fragmentation Table 2. Overview of Task Assignment Metrics ADJUSTING EFFORT ESTIMATION MODELS We believe that these three task assignment factors (concurrency, fragmentation, and intensity) have the potential to significantly enhance the predictive power of commonly used effort estimation models. A linear regression model augmenting the Intermediate COCOMO I model with the three metrics increased prediction accuracy from PRED(30%) of 31% to PRED(30%) of 49%. However, the usability of such a regression-based approach is limited. It would require an estimate of each of these three factors in addition to the large number of factors already collected for a firm s standard effort estimation model. This estimation is complicated by the fact that in the previously examined model, all three metrics are represented by ratio data, rather than the nominal data used in COCOMO, such as Development Flexibility ranging from very low to extra high. The usability of the task assignment metrics depends not only on their power to improve effort estimation, but also on their simplicity. In this light, we propose a set of simple multiplicative effort adjustment factors that can be used to augment the power of any effort estimation model with the task assignment metrics. While some predictive power is lost in the shift from ratio to nominal data, the resulting ease of application is tremendously attractive. The basic steps needed to augment any effort estimation model are as follows: 3 Currently located at the end of the manuscript. 4

6 Step 1: Over a period of time, collect data regarding the levels of intensity, concurrency, and fragmentation experienced within the organization. Step 2: Once norms for the three task assignment metrics have been established, begin to compare those experienced for a specific development effort to the organizational average. A project believed to fall in the lower quartile for a metric is classified as Very Low. A project believed to fall in the upper quartile for a metric is classified as Very High. Others projects are considered to be Typical with respect to that metric. Step 3: Apply each of the three multiplicative adjustment factors (shown in Table 3 below) to the effort estimation calculated using your standard model. If the USC COCOMO II software is used, there is an opportunity to incorporate these factors into the model as user-defined cost drivers. Step 4: Monitor the quality of the resulting estimates, and calibrate to your particular development environment. Effort Adjustment Factor Metric Very Low Typical Very High Intensity Concurrency Fragmentation Table 3: Proposed Task Assignment Effort Adjustment Factors Obviously, these factors are not independent. Intensity and fragmentation tend to be related to the number of tasks being juggled by individual team members. A very low level of intensity, coupled with a very high level of fragmentation, nearly doubles the estimated level of effort. Of course, the tendency in such cases might be to add additional team members, thus increasing the level of concurrency. When all three factors negatively impact the project, the total effort is scaled by nearly two-and-a-half times. On the other hand, when all three factors positively impact the project (high intensity, low concurrency, low fragmentation), the total effort estimation is scaled down by a factor of nearly one-half. WHAT RESULTS SHOULD YOU EXPECT? The data set from our original study [9] was used to establish the effort adjustment factors (EAFs) found in Table 3. First, the data set was divided in half; half (set 1) to be used in creating the EAFs, the other half (set 2) to be used in evaluating their impact on model performance. Second, the level of intensity, concurrency, and fragmentation associated with each task in set 1 was determined by comparison to the average of each metric within set 1. Linear regression (using the necessary log transform) was then used to estimate each multiplicative factor. Three, four, and five levels of each 5

7 metric were evaluated. The 4 and 5-level models were rejected because their added complexity failed to result in significant improvement in effort estimation. Once the EAFs were determined using set 1, they were evaluated using set 2 as shown in Table 4. Significant improvement in model performance was found for both COCOMO I and COCOMO II. While individual performance improvements will vary, results of this analysis are promising. COCOMO I prediction accuracy rose from a PRED(30%) of 29% to 35%. Similarly, the prediction accuracy of COCOMO II climbed from PRED(30%) of 37% to 43%. Intermediate COCOMO I MAP-Augmented COCOMO I Goodness of Fit COCOMO II Post Arch. MAP-Augmented COCOMO II Average Square Root Error Quality of Estimation PRED(30%) 29% 6 35% 37% 43% Table 4: Impact of Task Assignment EAFs CONCLUSION Our research has shown that software development effort decreases with: breaking work assignments down into tasks that can be accomplished individually; compressing the development schedule of tasks; and allowing teams to focus on a small number of tasks. Currently popular effort estimation models fail to take into account these task assignment factors. Rather than starting anew, we recommend adding multiplicative adjustment factors to reflect the impact of task assignment on development effort. We have shown significant improvement in predictive power by augmenting the popular COCOMO I and II models with effort adjustment factors to be applied when intensity, fragmentation, and/or concurrency vary dramatically from the norm. 4 In our original study, the Intermediate COCOMO I model was used for comparison of model performance. Since the completion of that study, the Post-Architecture COCOMO II model was applied and made available for analysis. 5 Varies from the ASRE found in Table 1 due to the use of only half of the data set. 6 Varies from the PRED (30%) found in Table 1 due to the use of only half of the data set. 6

8 REFERENCES 1. V. Basili and R. Reiter, An Investigation of Human Factors in Software Development, Computer, vol. 12, no. 12, pp , B. Boehm, Software Engineering Economics, Prentice-Hall, Inc., B. Boehm, B. Clark, E. Horowitz, C. Westland, R. Madachy, and R. Selby Cost Models for Future Software Life Cycle Processes: COCOMO 2.0, Annals of Software Engineering, vol. 1, pp , F. Brooks, The Mythical Man-Month, Addison Wesley Publishing, J. Cougar and H. Adelsberger, Comparing Motivation of Programmers and Analysts in Different Soci/Political Environments: Austria Compared to the United States, Computer Personnel, vol. 11, no. 4, pp , T. DeMarco and T. Lister, Peopleware: Productive Projects and Teams, Dorset House, S. Devani-Chulani, B. Clark and B. Boehm, Calibrating the COCOMO II Post-Architecture Model, Proceedings of the Twenty-second Annual Software Engineering Workshop, Software Engineering Laboratory, pp , N. Fenton and S. Pfleeger, Software Metrics: A Rigorous & Practical Approach, PWS Publishing Company, R. Smith, J. Hale and A. Parrish, An Empirical Study Using Task Assignment Patterns to Improve the Accuracy of Software Effort Estimation, IEEE Transactions on Software Engineering, in press. 7

9 Sidebar: Computing the Task Assignment Metrics We proposed methods for computing intensity, concurrency and fragmentation in [9]. All three metrics begin with the notion of a time unit, which is just an interval of time of fixed duration. In [9], we choose time units to have duration of one day, although other time unit sizes may be used (with different resulting values for intensity, concurrency and fragmentation). As an example, consider a task that is eight days (time units) in duration. Suppose that work is reported on this task by at least one developer on five of the eight days; for three of the eight days, no one reported work on the task. Then the intensity associated with this particular task is 5/8. With regard to concurrency, suppose that the number of developers reporting work on the task on each of the five days is given by the set {3,2,3,2,3}. Concurrency is computed as an average over the five days, i.e., ( )/5. With regard to fragmentation, the computation is slightly more complex. The following table represents the number of tasks for which each of the developers reported activity during each of time units when work was reported for task j. That is, t 1, t 3, t 4, t 7 and t 8 are time units when someone reported work on task j. For example, as the table shows during time unit t 1, developer p 1 reported work on three different tasks, developer p 5 reported work on two different tasks, and developer p 9 reported work on one task. Task j Developers p p p p t 1 t 3 t 4 t 7 t 8 Time Units Fragmentation for this particular task (j) is computed by averaging the number of tasks reported by each developer over the duration of task j. In particular, the sum is taken for each row ( , 2+1+1, 2+1+1, 1+2+2), and then divided over the number of team members (4). 8

10 AUTHOR BIOGRAPHIES Joanne E. Hale Joanne E. Hale is an Assistant Professor of Management Information Systems at The University of Alabama. She holds a Ph.D. in MIS from Texas Tech University, as well as an M.A. in statistics and a B.S. in industrial engineering from The University of Missouri. Her research interests include software cost estimation, component-based development and reuse, crisis support systems and small business IS strategy. Her previous research has appeared in IEEE Transactions on Systems, Man and Cybernetics, Journal of Software Maintenance, Journal of Management Information Systems, Information Resources Management Journal and Journal of Systems Management, and has been funded by organizations including Texas Instruments, Exxon, and the Center for Telecommunications Management. Allen Parrish Allen Parrish is an Associate Professor of Computer Science at The University of Alabama. He received a Ph.D. in computer and information science from The Ohio State University in His research interests principally include component-based software development and data warehousing. His current research sponsors include the National Science Foundation, NASA, the Alabama Department of Transportation and the Alabama Department of Economic and Community Affairs. Dr. Parrish is a member of the Association for Computing Machinery and the IEEE Computer Society, where he is a member of the Educational Activities Board. Brandon Dixon Brandon Dixon is an Associate Professor in the Department of Computer Science at The University of Alabama. He received a Ph.D. in computer science in 1993 from Princeton University. His interests are in computer science theory, algorithm design and software engineering. His current research sponsors include the National Science Foundation, NASA, the Alabama Department of Transportation and the Alabama Department of Economic and Community Affairs. Randy K. Smith Randy K. Smith is an Assistant Professor of Computer Science at Jacksonville State University. He received an M.S. and Ph.D. in Computer Science from The University of Alabama. His research interests include cost estimation modeling, component-based software development and distributed software engineering. 9