COCOTS: a COTS software integration cost model - model overview and preliminary data findings

Transcription

1 COCOTS: a COTS software integration cost model - model overview and preliminary data findings Chris Abts, Barry W. Boehm, and Elizabeth Bailey Clark Abstract As the use of commercial-of-the-shelf (COTS) components becomes ever more prevalent in the creation of large software systems, the need for the ability to reasonably predict the true lifetime cost of using such software components grows accordingly. In using COTS components, immediate short-term gains in direct development effort & schedule are possible, but usually as a trade-off for a more complicated long-term post-deployment maintenance environment. This paper discusses a COTS cost model being developed as an extension of the COCOMO II [1] cost model. COCOTS attempts to predict the lifecycle costs of using COTS components by capturing the more significant COTS risks in its modeling parameters. The current state of the model is presented, along with some preliminary findings suggested by an analysis of calibration data collected to date. The paper concludes with a discussion of the on-going effort to further refine the accuracy and scope of COCOTS. 1. Introduction COCOTS is the acronym for the COnstructive COTS integration cost model, where COTS in turn is short for commercial-off-the-shelf, and refers to those pre-built, commercially available software components that are becoming ever more important in the creation of new software systems. The rationale for building COTS-containing systems is that they will involve less development time by taking advantage of existing, market proven, vendor supported products, thereby reducing overall system development costs. But there are two defining characteristics of COTS software, and they drive the whole COTS usage process: 1) the COTS product source code is not available to the application developer, and 2) the future evolution of the COTS product is not under the control of the application developer. (Note: in some cases, COTS software does come as source code, but for the purposes of our model, if the developer does anything to that code other than compile it unchanged into his own code, we treat that as a reuse item and model its usage as adapted code within COCOMO II itself.) Because of these characteristics, there is a trade-off in using the COTS approach in that new software development time can indeed be reduced, but generally at the cost of an increase in software component integration work. The long term cost implications of adopting the COTS approach are even more profound, because you are in fact adopting a new way of doing business from the moment you start considering COTS components for your new system to the day you finally retire that system. This is because COTS software is not static, it continually evolves in response to the market, and you as the system developer must adopt methodologies that cost-effectively manage the use of those evolving components. 325

2 2. Model Description 2.1. Model Overview COCOTS at the moment is composed of four related submodels, each addressing individually what we have identified as the four primary sources of COTS software integration costs. (This is a key point. COCOTS as described in this paper deals only with initial integration efforts. We have already begun taking the first steps toward expanding the model to cover long-term lifecycle and maintenance costs associated with using COTS components. However, that topic will remain outside the scope of this paper.) Initial integration costs are due to the effort needed to perform (1) candidate COTS component assessment, (2) COTS component tailoring, (3) the development and testing of any integration or "glue" code (sometimes called glueware or binding code) needed to plug a COTS component into a larger system, and (4) increased system level programming and testing due to volatility in incorporated COTS components. (A fifth cost source was actually identified early in our research. This was the cost related to the increased verification and validation effort unrelated to COTS volatility but still usually required when using COTS components. But attempts have been made to capture these costs within the glue code and tailoring submodels directly rather than specify a fifth independent submodel.) Assessment is the process by which COTS components are selected for use in the larger system being developed. Tailoring refers to those activities that would have to be performed to prepare a particular COTS program for use, regardless of the system into which it is being incorporated, or even if operating as a stand-alone item. These are things such as initializing parameter values, specifying I/O screens or report formats, setting up security protocols, etc. Glue code development and testing refers to the new code external to the COTS component itself that must be written in order to plug the component into the larger system. This code by nature is unique to the particular context in which the COTS component is being used, and must not be confused with tailoring activity as defined above. Volatility in this context refers to the frequency with which new versions or updates of the COTS software being used in a larger system are released by the vendors over the course of the system's development and subsequent deployment. It should also be noted that while the assessment submodel lends itself easily to use very early in the project planning stages, the tailoring, glue code, and volatility submodels by the very nature of the costs they are trying to address are more problematic if used before candidate COTS products that may actually be integrated have been identified. The reason is that the costs covered by these models are extremely dependent on the unique characteristics of any given set of COTS products and their vendors Sources of Effort In terms of level of effort associated with our four primary COTS integration cost sources, the data we ve collected to date suggests that the greatest COTS integration effort tends to be concentrated on glue code development. One likely reason for this is that the writing of glue code is constrained in ways the writing of new code is not. Another reason may be that difficulties that must be addressed in the glue code often do not show up until the COTS integration process is well underway, making backtracking of designs and rework a necessity. The next most significant source of effort as suggested by our data tends to be tailoring. The fact that tailoring as we ve defined it is generally a lesser effort than glue code development seems intuitive. Tailoring activities can be involved, but are not usually overly 326

3 complex. They often follow templates and standard procedures. The writing of glue code on the other hand can require great creativity and ingenuity on the part of the programmer. LCO (requirements review) LCA (preliminary design review) IOC (system delivery) Staffing 1. COTS Assessment 2. COTS Tailoring 3. Glue Code Development New System Development Not Involving COTS Components 4. System Effort due to COTS Volatility Time LCO Lifecycle Objectives LCA Lifecycle Architecture IOC Initial Operational Capability COCOTS Effort Estimate COCOMO II Effort Estimate Figure 1. Sources of Effort Assessment tends to be the least significant source of COTS integration effort, though still measurably important. Again, however, that this would generally comprise the smallest effort overall is intuitive. Projects that get bogged down in the assessment phase are not making much progress towards delivery. (As an aside, however, as we ve collected our data, it has become apparent that the most successful projects have not given the assessment phase short shrift. They ve spent the effort necessary to verify that COTS products will do exactly what the vendors say they will do.) As for the level of effort associated with managing the COTS volatility effects in the larger system during initial development, it appears to be larger than that associated with assessment, but its true significance as reflected in our data is still unclear to us, and suggests that this may be one area where our model definition still needs further refinement. It also needs to be pointed out that the above relative effort orderings are only generalities. The true size of each effort that can be expected during a given software development is highly dependent on the design specifics of the given system. During data collection, we have come across projects in which only tailoring was done and no glue code was written, and vice versa. (No projects, however, got away without expending at least some effort on product assessment.) Figure 1 illustrates how the modelling of these effort sources in COCOTS is related to effort modelled by COCOMO II. The figure represents the total effort to build a software system out of a mix of new code and COTS components as estimated by a combination of COCOMO II and COCOTS. The central block in the diagram indicates the COCOMO II estimate, that is, the effort associated with any newly developed software in the system.. The smaller, exterior blocks indicate COCOTS estimates, that effort associated with the COTS 327

4 components in the system. The relative sizes of the various blocks in this figure are a function of the number of COTS components relative to the amount of new code in the system, and of the nature of the COTS component integration efforts themselves. The more complex the tailoring and/or glue code-writing efforts, the larger these blocks will be relative to the assessment block. Also, note that addressing the system-wide volatility due to volatility in the COTS components is an effort that will obtain throughout the entire system development cycle, as indicated by the long block running along the bottom of the figure. Finally, we caution the reader to not draw the erroneous conclusion that the various sources of effort being discussed can be as cleanly parsed as would appear to be indicated by the distinct boundaries drawn for each block. The reality is that rather than sharp lines, these boundaries would probably be more realistically shown as areas of different shading that blend into each other. This is particularly true for the boundary between the bottom volatility block and the larger system block Assessment Assessment is the activity whereby COTS software products are vetted and selected as viable components for integration into a larger system. In general terms, viable COTS candidates are determined based on the following:?? Functional Requirements - capability offered?? Performance Requirements - timing and sizing constraints?? Nonfunctional Requirements cost, training, installation, maintenance, reliability The thing to remember is that COTS assessment and requirements definition must be done in concert. Final system requirements must be shaped based upon the capabilities of COTS products on the market if one is to reap the full benefit of taking the COTS approach to system development. In the specific terms of our model, assessment is done in two passes. When selecting candidate COTS components, there is usually a quick and dirty first effort intended to very rapidly cull products patently unsuitable in the current context. The remaining products are then examined more carefully to determine the final set of COTS components to be included in the system being developed. Initial Filtering Effort (IFE)=? [(#COTS candidates in class)(average initial filtering effort for class)] over all classes Detailed Assessment Effort (DAE) =? [(#COTS candidates in class)(average detailed assessment effort for class)] over all classes, by project domain where the average detailed assessment effort for a given class is qualified by the attributes usually assessed for that class Final Project Assessment Effort = IFE + DAE 328

5 The key feature is that estimates are done based upon the classes of COTS components being examined. COTS components can usually be sorted by basic function such as GUI builders, operating systems, databases, word processors, etc. (Grouping COTS products into classes was also found by us to be the most effective way to gather calibration data. Managers and other data providers found it difficult to parse COTS integration data at either the individual component or system level, but found thinking in terms of classes of components a fairly straightforward way to ferret out the numbers for which we were looking.) The initial filtering effort formula presumes a parameterised value for a rough average filtering effort can be determined for a given class of COTS products. The Initial Filtering Effort (IFE) for a project then becomes a simple function of the number of candidate COTS products within a class being filtered for that project, the average filtering effort per candidate within that class of products, and the total number of COTS classes represented. The Detailed Assessment Effort (DAE) is similar. The parameterised average detailed assessment effort value, however, is qualified according to the kinds of attributes most often examined for a given a class of products when doing a product selection, and by the project domain (major area of application). Final selection of COTS products is typically based on an assessment of each product in light of certain product attributes. Table 1 contains a set of assessment attributes we ve found covers most assessment cases. (Adapted from IEEE Standards on Software Engineering.) Table 1 COTS Assessment Attributes Correctness Version Compatibility Price Availability/Robustness Intercomponent Maturity Compatibility Security Flexibility Vendor Support Product Performance Installation/Upgrade Ease Training Understandability Portability Vendor Concessions Ease of Use Functionality 2.4. Tailoring Tailoring is the activity whereby COTS software products are configured for use in a specific context. These are the normal things that would have to be done to a product no matter what system into which it is being integrated. These are things like parameter initialisation, input/gui screen and output report layout, security protocols set-up, etc. Specifically excluded from this definition is anything that could be considered a unique or atypical modification or expansion of the functionality of the COTS product as delivered by the vendor. (These activities would be covered under the glue code model.) Project Tailoring Effort =? [(#COTS tailored in class)(average tailoring effort for class and complexity)] over all classes, by project domain where the average tailoring effort for a given class is qualified by the complexity of the overall tailoring task usually associated with that class 329

6 The formulation of the tailoring submodel presumes that the difficulty or complexity of the tailoring work that is going to be required to get a given COTS product integrated into a system can be anticipated and even characterised by standardised rating criteria. It presumes a parameterised value for a rough average tailoring effort for each class of COTS component, within a given project domain, can be derived based on those standard complexity ratings. The total COTS tailoring effort for a project then becomes a function of the number of COTS products within a class being tailored for that project, the average tailoring effort per component within that class of products and at the given level of complexity, and the total number of COTS classes represented. To arrive at a complexity rating for a given tailoring job, we have defined a five-levelled scale, ranging from very low through nominal to very high. The scale, however, is really multi-dimensional, and requires examining individual tailoring activities, which affect the aggregate complexity of the job. Table 2 illustrates this concept. The first five cells in the first column under the heading "Tailoring Activities & Aids" identify the major activities that fall under our definition of COTS Tailoring, while the last cell refers to automated tools which may mitigate the difficulty of doing a given COTS tailoring job. Moving to the right across the other columns in the table you would find specific criteria (not shown) for rating the complexity of the individual activities represented by each row, or the utility of any available tailoring tools in the case of the last row. The last column at the far right of the table provides space for recording the point values associated with the rating given to each individual item identified in the first column on the far left. These individual point values are then summed to provide the total point score indicated in the extreme lower-right corner of the table. This total score is then used to characterise the overall complexity of the given COTS tailoring effort by using table 3, titled "Final Tailoring Complexity Scale." You determine where the point total falls on the scale shown in that table and from that identify the COTS tailoring effort complexity rating associated with that point total. Tailoring Activities & Aids Table 2 Tailoring Complexity Individual Activity & Tool Aid Complexity Ratings VL L N H VH Points Parameter Specification Script Writing I/O Report Layout GUI Screen Specification Security/Access Protocol Initialisation & Set-up Availability of COTS Tailoring Tools Total Points: Table 3 Final Tailoring Complexity Scale 5 7 pts 8 12 pts pts pts pts VL L N H VH 330

7 2.5. Glue Code Glue code is the new code needed to get a COTS product integrated into a larger system. It can be code needed to connect a COTS component either to higher level system code, or to other COTS components also being used in the system. Reaching consensus on just what exactly constitutes glue code has not always been easy. For the purposes of our model, we finally decided on the following three part definition: glue code is software developed inhouse and composed of 1) code needed to facilitate data or information exchange between the COTS component and the system or some other COTS component into which it is being integrated or to which it is being connected, 2) coded needed to connect or "hook" the COTS component into the system or some other COTS component but does not necessarily enable data exchange between the COTS component and those other elements, and 3) code needed to provide required functionality missing in the COTS component and which depends upon or must interact with the COTS component. Glue Code Effort = where A? [(size)(1+crevol)] B?? (effort multipliers)?? A = linear scaling constant?? Size = size of the glue code in source-lines-of-code or function points?? CREVOL = percentage rework of the glue code due to requirements change or volatility in the COTS products?? B = an architectural nonlinear scaling factor?? Effort multipliers = 13 multiplicative effort adjustment factors with ratings from very low to very high The formulation of the glue code submodel uses the same general form as does COCOMO, but defines a different set of effort multipliers. The model presumes that the total amount of glue code to be written for a project can be predicted and quantified (in either source lines of code or function points), including the amount of reworking that code will likely undergo due to changes in requirements or new releases of COTS components by the vendors during the integration period. Also presumed to be predictable are the broad conditions that will obtain while that glue code is being written--in terms of personnel, product, system, and architectural issues--and that these too can be characterised by standardised rating criteria. The model then presumes that a parameterised value for a linear scaling constant (in either units of person-months per source lines of code or person-months per function points) can be determined for a given domain (this is the constant A in the formula). Other parameterised values are also presumed to be determinable by domain: a non-linear scale factor that accounts for diseconomies of scale that can occur depending on how thoroughly the architecting of the overall system into which the COTS component is being integrated was conducted; and some thirteen effort adjustment factors that linearly inflate or deflate the estimated size of the glue code writing effort based upon a rating from very low to very high of specific project conditions. 331

8 The total Glue Code Effort (GCE) for a project then becomes a function of the amount (or size) of glue code to be written, its estimated percentage rework, the linear constant A, the rated non-linear architectural scale factor, and the individual rated effort adjustment factors. Table 4 lists the thirteen linear effort multipliers and one non-linear scale factor that currently appear in the model. Each of these factors has detailed definitions and rating criteria, and the interested reader is referred to the COCOTS website [5] for complete descriptions of these factors. Table 4. Glue Code Effort Multipliers Personnel Drivers 1) ACIEP - COTS Integrator Experience with Product 2) ACIPC - COTS Integrator Personnel Capability 3) AXCIP - Integrator Experience with COTS Integration Processes 4) APCON - Integrator Personnel Continuity COTS Component Drivers 5) ACPMT - COTS Product Maturity 6) ACSEW - COTS Supplier Product Extension Willingness 7) APCPX - COTS Product Interface Complexity 8) ACPPS - COTS Supplier Product Support 9) ACPTD - COTS Supplier Provided Training and Documentation Application/System Drivers 10) ACREL - Constraints on Application System/Subsystem Reliability 11) AACPX - Application Interface Complexity 12) ACPER - Constraints on COTS Technical Performance 13) ASPRT - Application System Portability Nonlinear Scale Factor AAREN - Application Architectural Engineering 2.6. System Volatility System Volatility Effort (SVE) refers to that extra effort which occurs during the development of the larger application as a result of the use of COTS components in that system development. Specifically, it is the effort that results from the impact on the larger system of the effects of swapping COTS components out of the system with newer versions of those components that have been released by the COTS vendors. Over a lengthy initial development these impacts can be significant. Once the project has moved into the long-term maintenance phase, these impacts will be significant, and may in fact dominate your system maintenance processes, depending upon the number and nature of the COTS components you have used in your system. The volatility submodel tries to capture these effects by again borrowing from COCOMO the concept of rework due to factors other than errors in coding. In COCOMO, this is defined to be the amount of rework that must be done to code not because of errors in the initial programming, but rather because of changes in system requirements. In the glue code submodel, we added the additional qualifier of rework that must be done not just due to requirements change, but also as a result of integrating upgraded versions of a COTS component. This gave us the term CREVOL. 332

9 The volatility submodel considers two other kinds of rework: that in the larger system application code due to integration of updated COTS software versions (SCREVOL), and that in the application code independent of COTS product effects. (This latter rework is actually the same as defined within COCOMO itself and captured in the COCOMO REVL, or requirements evolution term.) System Volatility Effort = where (application effort)? {[1+(SCREVOL/1+REVL)] E 1}? (COTS effort multipliers)?? Application effort = new coding effort separate from COTS integration effects?? SCREVOL = percentage rework in the system due to COTS volatility and COTS requirements change?? REVL = percentage rework in the system independent of COTS effects due to requirements change?? E = ? (COCOMO Scale Factors)?? COTS effort multipliers = 13 multiplicative effort adjustment factors from the glue code submodel The application effort term appearing in this equation is that effort that would be modeled directly by COCOMO II. The effort adjustment factors, however, are those that appear in the COCOTS glue code submodel. Finally, the scale factor term appearing in the equation are the COCOMO II scale factors (Precedentedness, Development Flexibility, Architecture/Risk Resolution, Team Cohesion, and Process Maturity for a complete discussion of these terms, see reference [4]) Total COTS Integration Effort At the moment, COCOTS is treating this as the linear sum of the four sources of effort: Total COTS Integration Effort = Assessment Effort + Tailoring Effort + Glue Code Effort + System Volatility Effort As an approximation, this works for now, but the true relationship between these terms is likely more complicated. 3. References [1] Boehm, B., Clark, B., Horowitz, E., Madachy, R., Shelby, R. and Westland, C., Cost Models for Future Software Lifecycle Processes: COCOMO 2.0, in Annals of Software Engineering, [2] Adapted from Boehm, B. and Abts, C., COTS Integration: Plug and Pray?, Computer, Jan. 1996, pp [3] Boehm, B. Software Engineering Economics, Prentice Hall, NJ, [4] Boehm, B., Abts, C., Brown, A., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D. and Steece, D., Software Cost Estimation with COCOMO II, Prentice Hall, June [5] COCOTS website: 333

10 334