An Empirical Study of the Efficacy of COCOMO II Cost Drivers in Predicting a Project s Elaboration Profile

Transcription

1 An Empirical Study of the Efficacy of COCOMO II Cost Drivers in Predicting a Project s Elaboration Profile Ali Afzal Malik, Barry W. Boehm Center for Systems and Software Engineering University of Southern California Los Angeles, CA 90089, U. S. A. alimalik@usc.edu, boehm@usc.edu Abstract. A project s elaboration profile consists of a set of ratios called elaboration factors that quantify the step-wise expansion of a project s requirements from very high-level business objectives to very low-level source lines of code. Knowledge of a project s elaboration profile can be extremely useful in deriving an early estimate of its size. The real challenge, however, is to predict the elaboration profile. Can the COCOMO II cost drivers be used for this purpose? This paper attempts to answer this question. It examines the elaboration profiles and COCOMO II cost driver ratings of 25 small real-client projects. The data collection process is thoroughly described and the cost drivers relevant at each stage of elaboration are identified. Relationships between elaboration factors and relevant cost drivers are analyzed using simple as well as multiple regression. The results indicate that there is no magical formula for predicting the various elaboration factors just from the values of COCOMO II cost drivers. This may be due to some confounding factors which are highlighted in this paper. Keywords: COCOMO II cost driver, elaboration factor, elaboration profile, empirical study, requirements elaboration, software sizing 1. Introduction The requirements of a software development project undergo a step-wise refinement process called requirements elaboration. Very high-level business goals are refined into shall statements. Shall statements are elaborated into use cases. Use cases are expanded into use case steps. Finally, use case steps are implemented as source lines of code (SLOC). Requirements elaboration can be quantified by metrics called elaboration factors. These are ratios that measure the magnitude of expansion of high-level requirements into low-level ones. For instance, the elaboration factor between a project s business goals and shall statements indicates the average number of shall statements derived from a single goal of that project. The set of all elaboration factors of a project defines its elaboration profile. Since a project s requirements can be expressed at multiple levels of detail e.g. goals, shall statements, use cases, etc., each pair of consecutive requirements levels has an elaboration factor associated with it. Therefore, a project with n requirements levels will have n - 1 elaboration factors in its elaboration profile.

2 As shown in [Malik et al. 2009], knowledge of a project s elaboration profile can be extremely helpful in estimating its size at the time of inception. Elaboration profiles, however, are not available at the time of inception. They must be predicted. The real question, therefore, is whether or not it is possible to predict a project s elaboration profile? This paper attempts to answer this question by examining the relationship between the elaboration profile of a project and its COCOMO II [Boehm et al. 2000] cost drivers. The remaining paper is organized as follows. Section 2 presents the empirical setting of this study while Section 3 describes the process for determining the COCOMO II cost driver ratings. Data analysis results are reproduced in Section 4 along with a commentary on the salient aspects of these results. Section 5 concludes this paper with a discussion of the possible reasons for these results. 2. Empirical Setting This work analyzes the elaboration profiles and COCOMO II cost driver values of 25 small e-services projects [Projects Archive 2010] done by graduate students studying software engineering at USC. It is a continuation of our previous work [Malik et al. 2009] which examined the requirements-to-code elaboration factors of these projects. A list of these 25 projects is reproduced in Table 1. Table 1. Projects used by empirical study S# Year Project Name Project Type Online Bibliographies on Chinese Religions Web-based Database Data Mining of Digital Library Usage Data Stand-alone Analysis Tool Data Mining from Report Files Stand-alone Analysis Tool Data Mining PubMed Results Stand-alone Analysis Tool USC Football Recruiting Database Web-based Database Template-based Code Generator Stand-alone Conversion Tool Web-based XML Editing Tool Web-based Editor EBay Notification System Stand-alone Analysis Tool Rule-based Editor Stand-alone Database CodeCount Product Line with XML and C++ Stand-alone Analysis Tool USC Diploma Order and Tracking System Web-based Database Student's Academic Progress Application Web-based Database USC CONIPMO Stand-alone Cost Model USC COINCOMO Stand-alone Cost Model BTI Appraisal Projects Web-based Database LAMAS Customer Service Application Web-based Database BID Review System Web-based Database

3 Proctor and Test Site Tracking System Web-based Database Master Pattern Web-based Database Housing Application Tracking System Web-based Database EZBay Stand-alone Analysis Tool AAA Petal Pushers Remote R&D Web-based Database Online Peer Review System for Writing Program Web-based Database The Virtual Assistant Living and Education Program Stand-alone Database Theatre Script Online Database Web-based Database As shown in Table 1, these projects were undertaken between 2004 and Projects begun in 2004 followed the MBASE/RUP [Boehm et al. 2004, Kruchten 2003] software development process while those initiated between 2005 and 2007 followed its leaner version i.e. LeanMBASE/RUP [Boehm et al. 2005, Kruchten 2003] projects followed the Instructional ICM-Sw [Instructional ICM-Sw 2010] process. A variety of domains and types are represented by these projects. Over half of them, however, are web-based databases. Clients of these projects were mostly USC neighborhood organizations (e.g. Manual Arts High School, Los Angeles Music and Art School, etc.), USC CSSE affiliates (e.g. Northrop Grumman, The Aerospace Corporation, etc.), or USC departments (e.g. Anthropology, Degree Progress, Library, etc.). Even though each project was unique, these projects had a lot in common. First and foremost, each project was completed in two semesters. Using schedule as an independent variable (SAIV) implied that requirements had to be prioritized to facilitate timely completion. Secondly, each project was done by teams of 6 8 students. Usually, two members of the team were responsible for independent verification and validation (IV & V) with the majority forming the development core of the team. The IV & V members were off-campus students with an average of at least five years of industry experience. The development core consisted of on-campus students with an average industry experience of a couple of years. Thirdly, each project was closely monitored by the teaching staff for quality and timely completion of deliverables. Furthermore, each project used industry grade processes and tools. Last but not the least, each project was a custom-built project. In other words, COTS-based projects were not considered. 3. Method The generic process for determining the complete elaboration profile of a project has already been described in our previous work [Malik et al. 2009]. Our previous work also presents the elaboration profile of each of the 25 projects considered here. This section, therefore, focuses on describing the process used for collecting the COCOMO II driver values.

4 Table 2. Cost driver rating scale Rating Level Abbreviation Value Very Low VLO 1 Low LO 2 Nominal NOM 3 High HI 4 Very High VHI 5 Extra High XHI 6 Table 2 displays the rating scale used for COCOMO II cost drivers. It displays the abbreviations of rating levels along with the numeric value assigned to each of the six rating levels. Since increments between rating levels are also possible, the final rating of a cost driver is determined by the following equation: Equation 1: Rating = Value + Increment% If a cost driver is rated, for instance, at VLO with an increment of 50% then its numeric rating will be = 1.5. If there is no increment (i.e. Increment% = 0), which is most often the case, Rating is simply equally to the numeric Value obtained using the mapping given in Table 2. Information about the rating levels of COCOMO II cost drivers is obtained from the final version of a project s Life Cycle Plan (LCP). The final version of the LCP is usually present in the last Initial Operational Capability (IOC) working set, as-built set, or the final deliverables of the project. The rationale for choosing the final version of LCP is two-fold. Firstly, the elaboration profiles were obtained using documents produced at the IOC anchor point milestone [Boehm 1996] which maps to the end of the construction phase. Thus, in order to ensure consistency with the elaboration profiles, COCOMO II cost driver ratings are also derived from the same milestone. Secondly, it is important to use stabilized ratings of cost drivers. The final version of the LCP contains ratings that have stabilized after evolution over the course of the project. Finding the numeric ratings of a project s five scale factors PREC, FLEX, RESL, TEAM, and PMAT is a straight forward process. This is primarily because these scale factors apply to the project as a whole. Once the rating levels of these scale factors have been obtained from the LCP, all that needs to be done is to map the rating levels to the numeric values using Table 2 and then use Equation 1 to get the final rating. Even though the multiplicative SCED cost driver also applies to the project as a whole, it can be easily omitted from this particular analysis. This is because all 25 projects examined in this study use the SAIV process model and thus have the same value (i.e. NOM) for the SCED cost driver. Determining the numeric ratings of the multiplicative cost drivers (excluding SCED) requires more effort. The reason for this is that these multiplicative cost drivers apply to the individual modules of the project. In other words, each module of the project has its own separate list of multiplicative cost driver ratings. In order to reconcile this

5 difference between the applicability of project-wide scale factors and module-wide multiplicative cost drivers, module-wide cost drivers are weighted by module size to come up with a project-wide rating. The following formula is used for this conversion: Equation 2: Pr ojectrating n ModuleRating ModuleSize i= 1 = n i= 1 i ModuleSize In Equation 2, n represents the number of modules in the project. Modules that are bigger in size have more impact on the ProjectRating vis-à-vis smaller modules. Consider, for instance, a project that has three modules M1, M2, and M3 with sizes 100 KSLOC, 200 KSLOC, and 300 KSLOC respectively. If CPLX rating levels for M1, M2, and M3 are LO, NOM, and HI respectively, then using Table 2 (and assuming Increment% = 0) the overall project CPLX rating is given by: 4. Results (2 100) + (3 200) + (4 300) = ( ) 2000 = Table 3 summarizes the COCOMO II cost driver ratings for each of the 25 small realclient projects. These ratings have been calculated using the procedure described in the previous section. For ease of reference, the first column S# - maps to the first column of Table 1. i i Table 3. COCOMO II cost driver ratings Scale Factors Product Factors Platform Factors Personnel Factors Project Factors PREC FLEX RESL TEAM PMAT RELY DATA CPLX RUSE DOCU TIME STOR PVOL ACAP PCAP PCON APEX PLEX LTEX TOOL SITE S#

6 As a first step towards determining the utility of COCOMO II cost drivers in predicting the elaboration profile of a project, a simple regression analysis is carried out. UC The individual correlation between each of the 4 elaboration factors (i.e.,, CG UCS SLOC, and ) and each of the 21 COCOMO II cost drivers (i.e. 5 scale factors and UC UCS 16 multiplicative cost drivers) is determined. Needless to say, in all of these regression analyses, the elaboration factor acts as the response variable while the cost driver acts as the predictor. All in all, 4 x 21 = 84 correlations are calculated. Tables 4 through 7 summarize the results of these simple regression analyses. These tables have been sorted in descending order of the correlation strength. In other words, COCOMO II drivers having a higher value of the coefficient of determination (R 2 ) appear above those with lower values. The first column of these tables Rank indicates the rank of the cost driver with respect to its correlation strength. The column labeled +/- specifies the direction of the relationship between the elaboration factor and the cost driver as indicated by the sign of the slope coefficient of the fitted mean function using ordinary least squares. A + indicates a positive relationship while a - signifies a negative or inverse relationship. The last column Type displays the category of the COCOMO II cost driver i.e. scale factor, product factor, platform factor, personnel factor, or project factor. Table 4. Correlation between and COCOMO II cost drivers CG Rank R 2 +/- COCOMO II Driver Type FLEX Scale Factor ACAP Personnel Factor PMAT Scale Factor SITE Project Factor PCAP Personnel Factor RELY Product Factor RESL Scale Factor

7 DATA Product Factor CPLX Product Factor PLEX Personnel Factor STOR Platform Factor APEX Personnel Factor TIME Platform Factor DOCU Product Factor PREC Scale Factor PVOL Platform Factor PCON Personnel Factor RUSE Product Factor LTEX Personnel Factor TOOL Project Factor E-06 + TEAM Scale Factor UC Table 5. Correlation between and COCOMO II cost drivers Rank R 2 +/- COCOMO II Driver Type TEAM Scale Factor PMAT Scale Factor SITE Project Factor TOOL Project Factor LTEX Personnel Factor PLEX Personnel Factor RESL Scale Factor DOCU Product Factor PVOL Platform Factor PCAP Personnel Factor STOR Platform Factor ACAP Personnel Factor CPLX Product Factor APEX Personnel Factor PCON Personnel Factor FLEX Scale Factor PREC Scale Factor

8 RELY Product Factor RUSE Product Factor DATA Product Factor TIME Platform Factor UCS Table 6. Correlation between and COCOMO II cost drivers UC Rank R 2 +/- COCOMO II Driver Type PREC Scale Factor SITE Project Factor TEAM Scale Factor APEX Personnel Factor DOCU Product Factor PMAT Scale Factor PLEX Personnel Factor STOR Platform Factor RESL Scale Factor LTEX Personnel Factor PCAP Personnel Factor DATA Product Factor TOOL Project Factor RELY Product Factor CPLX Product Factor PVOL Platform Factor PCON Personnel Factor FLEX Scale Factor RUSE Product Factor TIME Platform Factor E-05 - ACAP Personnel Factor Table 7. Correlation between SLOC and COCOMO II cost drivers UCS Rank R 2 +/- COCOMO II Driver Type PCON Personnel Factor

9 PMAT Scale Factor RELY Product Factor TOOL Project Factor STOR Platform Factor TEAM Scale Factor CPLX Product Factor RUSE Product Factor PVOL Platform Factor PCAP Personnel Factor PREC Scale Factor DOCU Product Factor APEX Personnel Factor TIME Platform Factor RESL Scale Factor PLEX Personnel Factor DATA Product Factor FLEX Scale Factor ACAP Personnel Factor LTEX Personnel Factor SITE Project Factor Table 4 shows the simple correlation between the first elaboration factor - - CG and each of the 21 cost drivers. represents the elaboration of capability goals (CG) CG into capability requirements (). While all correlations are weak, and FLEX have CG the strongest correlation i.e UC UC Correlations of cost drivers with are shown in Table 5. is the second elaboration factor which measures the elaboration of capability requirements () into UC use cases (UC). has the highest correlation with TEAM (0.4707) followed by PMAT (0.2694), SITE (0.1531), TOOL (0.1425), and LTEX (0.1233). Correlations with the rest of the cost drivers are weak.

10 UCS The third elaboration factor - - quantifies the elaboration of use cases (UC) UC UCS into use case steps (UCS). Table 6 displays the correlations of cost drivers with. UC All correlations are weak. However, correlations with PREC (0.1148) and SITE (0.1016) are relatively stronger. Table 7 presents the correlations of cost drivers with the last elaboration factor i.e. SLOC measures the elaboration of use case steps (UCS) into source lines of SLOC. UCS UCS code (SLOC). In this final stage of elaboration, PCON (0.2265) and PMAT (0.2058) are the only cost drivers which have correlations worth mentioning. In the second step, multiple regression analysis (which uses one response and multiple predictors) is used to determine the utility of COCOMO II cost drivers in predicting the elaboration profile of a project. This involves identifying the cost drivers pertinent to each stage of elaboration. This list of relevant cost drivers is presented in Table 8. In this table, each stage is assigned a number for later reference and is labeled using the Cockburn altitude metaphor [Cockburn 2001]. Table 8: COCOMO II cost drivers relevant at each elaboration stage Elaboration Stage (Cockburn Metaphor) Stage 1: Capability Goals to Capability Requirements (Cloud to Kite) Stage 2: Capability Requirements to Use Cases (Kite to Sea Level) Stage 3: Use Cases to Use Case Steps (Sea Level to Fish) Stage 4: Use Case Steps to SLOC (Fish to Clam) Relevant Cost Drivers ACAP, APEX, DOCU, FLEX, PREC, RESL, SITE, TEAM ACAP, APEX, PCON, PREC ACAP, APEX, CPLX, DOCU, PCON, PREC APEX, CPLX, LTEX, PCAP, PCON, PLEX, PREC, PVOL, RELY, RUSE, STOR, TIME, TOOL At each stage of elaboration, cost drivers are short-listed based on their perceived applicability at that stage. Applications experience and precedentedness can influence elaboration at each stage thus APEX and PREC are listed as potential predictors in all four stages. Personnel continuity (PCON) is only relevant after the first stage of elaboration. Thus, PCON appears in all stages except the first. Required software reliability (RELY) deals with level-of-service (LOS) or non-functional requirements and is, therefore, not considered at all. Process maturity (PMAT) is omitted from all four stages since each of the 25 projects is done by teams of graduate students that are approximately at the same level of process maturity. Similar reasoning is used to shortlist the other cost drivers. As a rule of thumb, scale factors are more relevant in earlier

11 stages of elaboration while product factors, platform factors, and project factors are more relevant at later stages. The relevance of personnel factors varies from stage to stage. Programmer capability (PCAP), for instance, is only relevant at the last stage of elaboration which deals with SLOC. Analyst capability (ACAP), on the other hand, is relevant in all stages except the last. Table 9 shows the results of the multiple regression analyses conducted using the short-listed predictors shown in Table 8. The low values of the multiple correlation coefficients (multiple R 2 ) indicate very weak predictive power of the short-listed cost drivers. Moreover, the low significance levels (high p-values) signify very strong evidence in favor of the null hypothesis that the means of the elaboration factors do not depend on any of their respective short-listed cost drivers. Table 9: COCOMO II cost drivers relevant at each elaboration stage Elaboration Stage Multiple R 2 P-value Stage 1: Capability Goals to Capability Requirements Stage 2: Capability Requirements to Use Cases Stage 3: Use Cases to Use Case Steps Stage 4: Use Case Steps to SLOC Discussion Results reported in the previous section clearly imply that there is no magical formula for predicting the elaboration profile of a project using only the information contained in the COCOMO II cost drivers. Correlations obtained using the simple and multiple regression analyses are weak indicating that the variance of cost drivers cannot adequately explain the variance in elaboration factors. This weak predictive power of the COCOMO II cost drivers may be due to a number of confounding factors. First and foremost, the accuracy of data needs to be considered. [Malik et al. 2009] found that it was necessary to normalize multi-level requirements data in order to obtain consistent elaboration profiles. Therefore, it may be necessary to normalize COCOMO II cost driver ratings across these 25 projects. In fact, concrete evidence already points in favor of this. While obtaining the cost driver ratings, it was observed that different teams had assigned different rating levels for PCON even though they had cited the same rationale. Similarly, even though all student teams were approximately at the same level of process maturity, a number of different rating levels were noticed for PMAT. Such occurrences clearly indicate the need for having a list of guiding principles to assist student teams in selecting cost drivers rating levels. This would help a great deal in obtaining consistent ratings of cost drivers across different projects. Another potential confounding factor is the students incentive structure. Unlike the individuals involved in industry projects, team members of projects undertaken in academia are motivated by the desire to get good grades in each deliverable. A natural consequence of this is over-specification of information. For instance, teams may

12 document the capability goals in the Operational Concept Description document at the level of detail of capability requirements. This practice easily corrupts the elaboration data. Last, but certainly not the least, the variation amongst the clients of these projects may have played a significant role in the results obtained. As found out by interviewing the teaching assistants responsible for grading these projects, not all clients wanted quick feedback in the form of prototypes and screen shots. The demand for quick feedback may also lead to an over-specification of information at higher levels of requirements. This, in turn, spoils the elaboration data. The same interview also asked the teaching assistants to rank the clients on the ACKD collaborative, representative, authorized, committed, knowledgeable, and decisive characteristics using a scale of 1 (least) to 5 (most). ACKD is a slightly augmented version of the ACK customer characteristics mentioned in [Boehm and Turner 2004]. Decisiveness measures the ability of the client to state requirements that are final and not subject to later modification. As expected, each client had his/her own ACKD profile. These client characteristics may also be responsible for some of the variation in the elaboration profiles. A client that is more representative and knowledgeable, for instance, may provide more information upfront leading to lower elaboration factors at higher levels of requirements. On the other hand, a project with a less decisive client may experience greater elaboration in the earlier stages. 6. Acknowledgements The authors thank Supannika Koolmanojwong and Pongtip Aroonvatanaporn the teaching assistants for USC software engineering courses CSCI 577A and CSCI 577B for their help in providing information related to the clients of the projects analyzed in this study. 7. References Boehm, B. (1996). Anchoring the Software Process, IEEE Software 13(4), pages Boehm, B., Abts, C., Brown, A., Chulani, S., Clark, B, Horowitz, E., Madachy, R., Reifer, D., and Steece, B. (2000), Software Cost Estimation with COCOMO II, Prentice Hall. Boehm, B., Brown, A., Port, D., et al. (2004). Guidelines for Model-Based (System) Architecting and Software Engineering (MBASE), Center for Software Engineering, University of Southern California. Boehm, B., Klappholz, D., Colbert, E., et al. (2005). Guidelines for Lean Model-Based (System) Architecting and Software Engineering (LeanMBASE), Center for Software Engineering, University of Southern California. Boehm, B. and Turner, R. (2004), Balancing Agility and Discipline: A Guide for the Perplexed, Addison-Wesley. Cockburn, A. (2001), Writing Effective Use Cases, Addison-Wesley. Instructional ICM-Sw (2010). Instructional Incremental Commitment Model-Software Electronic Process Guide (EPG),

13 Kruchten, P. (2003), The Rational Unified Process: An Introduction, Addison-Wesley. Malik, A., Koolmanojwong, S., and Boehm, B. (2009). An Empirical Study of Requirements-to-Code Elaboration Factors, 24th International Forum on COCOMO and Systems/Software Cost Modeling. Projects Archive (2010).