Position Paper for the International Workshop on Reuse Economics, Austin, Texas

Transcription

1 Position Paper for the International Workshop on Reuse Economics, Austin, Texas COTS-based Systems and Make vs. Buy Decisions: the Emerging Picture Chris Abts Information & Operations Management Department Texas A&M University and Center for Software Engineering University of Southern California Abstract. The conventional rationale for using COTS (commercial off-theshelf) components is that the more a software system is built from COTS products, the lower the cost of initial development. Less understood is that during the long term sustainment phase from deployment through retirement the cost of maintenance of a COTS-based system (CBS) generally increases as the number of COTS products used increases. There exists then a tension between the imperative to maximize the use of COTS components to ease CBS development yet minimize the use of COTS components to ease CBS maintenance. These apparently conflicting views are reconciled in a design heuristic that has been previously proposed called the CBS Functional Density Rule of Thumb. However, a better understanding of these phenomena needs to be established before truly honest make vs. buy decisions can be had. The key is that costs over the entire system lifecycle must be examined when doing a make vs. buy analysis. This in itself is not news. My position, however, is that to date, the true long-term cost implications of using COTS components within a system have been poorly understood, particularly the cost of using many COTS components from multiple vendors. This has likely led in many cases to a tipping of the scales in favor of purchased solutions over custom solutions based on inaccurate assumptions regarding the maintenance and sustainment costs of COTS-based vs. custom systems. A better understanding of COTS maintenance costs might in fact have made the custom solution more favorable in at least some of these instances. Therefore I believe that allocating significant research resources to exploring further CBS maintenance phenomena would be worthwhile.

2 1 CBS Economics - The Conventional View One of the most significant approaches widely adopted in the last two decades that absolutely represents a systemic change in the way large software systems are created is the now widespread use of commercial-off-the-shelf (COTS) components as system elements. It is likely that not a single large scale software system on the drawing boards today is being designed without the incorporation of at least one COTS component and probably several. Contrary to what a few may have once believed, however, when compared to building from scratch, COTS solutions aren't necessarily free. Less than a decade ago, there was perhaps an overeager hope among some software procurers that the COTS approach to building software systems might indeed be if not Brooks' magic bullet [1] then at least a very strong prescriptive for containing the software beast. The closer one got to the executive levels far removed from the nuts-and-bolts grind of getting COTS solutions to work, the stronger this hope seems to have become. Policies were established strongly favoring COTS-based systems over in-house development [2]; contracts were reportedly let mandating required levels of COTS supplied functionality; the prevailing attitude was "the more COTS components the better, since it means the less we have to build ourselves." $ Cost of Initial Development Ratio of COTS Components vs. Original Components in System % Fig. 1. The conventional view of CBS economics The flaw in that logic was in not realizing that "not building" did not mean "not developing," only developing differently. The COTS approach requires effort be spent on tasks different from those that would be encountered if building from scratch, but these new tasks are not necessarily less time consuming. These include assessing and verifying purported capabilities of COTS products via extensive testing (both

3 individually and in combinations of components that must work together), tailoring the components to work in the specific context needed, and creating binding-ware or glue code to plug the components into the overall system. There are also a host of other tasks that must be tackled, including COTS product licensing and vendor management, training of both system developers and end-users, addressing issues of legal liability exposure, etc. Due to the hard-earned wisdom gained by early adopters of the COTS approach and which is now being shared with others [3,4], it is better understood today that the most frequently encountered scenario when adopting the COTS approach is that effort is time-shifted rather than significantly reduced. That is, initial and prototype systems constructed with COTS elements can often be developed and fielded more quickly than systems built entirely from scratch, but final system testing and post-delivery maintenance and long term sustainment can be highly challenging. The conventional view of the economic benefits of using the COTS approach to software system development is shown then in Figure 1. What is missing from this picture is a fully-realized understanding of the significant work that can be required to select, install and configure COTS components that provide the functionality needed as well as ensuring that the components will operate as intended together as a group. 2 CBS Economics - The COTS-LIMO View There is an alternative view towards the economics of COTS-based software systems to the one discussed above. Anecdotal evidence collected during data collection interviews performed to gather calibration data for the COCOTS [5] COTS integration cost estimation model extension of COCOMO II [6] suggests that generally though granted not universally the more COTS software components you include in your overall software system, the shorter the economic life will be of that system, particularly if doing present-worth analyses comparing alternative designs using various combinations of COTS components, or when comparing COTS-based designs to simply building the entire system from scratch. The reason is due to the volatility of COTS components. Volatility in this case means the frequency with which vendors release new versions of their products and the significance of the changes in those new versions (i.e., minor upgrades vs. major new releases). When you first deploy your system, you of course have selected a suite of components that will provide the functionality you require while at the same time work in concert with each other. Over time, however, those products will likely evolve in different directions in response to the market place, in some cases even disappearing from the market all together. As a consequence, the ability of these diverging products to continue functioning adequately together if and when you install the newer versions will likely also become more problematic; the more COTS components you have, the more severe the consequences of these increasing incompatibilities will become.

4 These ideas are expressed in something called the COTS-LIMO (COTS-Life span Model) which can be seen in Figure 2 [7]. As is seen in the figure, the graph is broken into two regions bisected by the line n. As long as the number of COTS components in the system is less than n, the increase in experience gained by your system maintainers over time and thus the inherent improvements in their productivity will outpace the increased effort required to maintain the system as the COTS products it contains age and evolve in divergent directions. However, at some number of installed COTS components n, the breakeven point is surpassed and no matter how skilled and experienced your maintainers become, the increases in their efficiency at maintaining the system can no longer keep pace with the impact of the increasing incompatibilities arising in the evolving COTS components. At this point you have reached the zone in which the useful life of your system has been shortened considerably and a decision to retire the system will soon have to be made. The actual value of n, the specific shape of the individual contour lines, and the location of the M-R (maintain-retire) line will be highly context sensitive, differing for each software system under review. Also, even though the model as shown uses the raw number of COTS components as the primary decision variable, in fact this is really just a surrogate for some measure of the complexity of the interfaces between the various COTS items in the system. In other words, a CBS with a lot of COTS components but which all have very simple and stable interfaces might still have a longer economic life span than a CBS with fewer COTS components but which all have very complex and volatile interfaces. $ n+x n+2 n+1 Number COTS in System Retire n (maintenance equilibrium value) Cost of Maintenance 3 2 M-R Line Maintain 1 Time t Fig. 2. The COTS-LIMO view of CBS economics

5 3 CBS Functional Density Rule So how can these opposing points of view be reconciled? The answer lies not in increasing the number of COTS components that you use, but rather in increasing the percentage of system functionality that you deliver via COTS components. This is a subtle but important change of perspective that suggests the following CBS design rule: The CBS Functional Density Rule of Thumb [8] Maximize the amount of functionality in your system provided by COTS components but using as few COTS components as possible. Corollary to the CBS Functional Density Rule The absolute number of COTS components in your system should not exceed the maintenance equilibrium value. (The maintenance equilibrium value is represented by n in Figure 2.) $ Total System Cost Over Life Cycle COTS Functional Density (as long as number COTS < n) CFD Fig. 3. Suggested Potential total life cycle cost profile of a CBS design adhering to the CBS Functional Density Rule

6 LCO 4 CBS Maintenance Costs the Emerging View The CBS total life cycle cost profile suggested in Figure 3 is yet to be demonstrated empirically. The current state of understanding of CBS maintenance issues is more akin to the diagram shown below: IOC RRR Retirement of System? R? R? R Staffing A T GC?? GC GC GC T T? A T A A COCOMO II COCOMO Maintenance Model V Volatility Transition TR TR TR TR Operations Time start cycle? repeating refresh cycles? end cycle? Development Transition Maintenance Fig. 4. CBS Post-deployment modeling horizon As shown in Figure 4, it is known that over time costs associated with assessing, tailoring, the writing of glue code, and managing the volatility of COTS components obtain during the initial development phase and then tend to repeat to varying degrees during periodic maintenance cycles through system retirement. At times there may even be costs associated with completely replacing specific COTS components. Other potential CBS maintenance costs not directly shown in Figure 4 have also been identified [9]. These are listed below as activities prioritized in terms of their criticality and/or the relative effort needed to perform them. They are also identified as being continuous costs throughout maintenance or as costs that tend to spike around the repeating refresh cycle transition period milestones.

7 Priority of CBS Maintenance Phase Activities by Effort Involved and/or Criticality Higher training S C configuration management C operations support C integration analysis S requirements management S C Medium certification S market watch C distribution S vendor management C business case evaluation S Lower administering COTS licenses C S - activity spikes around refresh cycle anchor points C - activity is continuous throughout maintenance SC - activity is continuous yet also spikes around cycle anchor points 5 Conclusions and Recommendations At best only a partial understanding is what we have currently regarding the true costs over time of using COTS components. To do a comprehensive, side-by-side make vs. buy comparison of a custom-built vs. a COTS-based system from initial proposal all the way through system retirement much more investigation must be performed to identify and characterize all of the most significant CBS maintenance costs. My position is that to date, the true long-term cost implications of using COTS components within a system have been poorly understood. This has likely led in many cases to the wrong choice being made when selecting between COTS-based and custom solutions where the deciding metric has been total system life cycle costs. Therefore I believe that allocating significant research resources to further exploring CBS maintenance phenomena is warranted. References 1. Brooks, F.P., Jr., No Silver Bullet: Essence and Accidents of Software Engineering, Computer, IEEE Computer Society, Washington, D.C., April Abts, C. and Boehm, B., COTS Software Integration Cost Modeling Study, USC-CSE tech. Report , USC Center for Software Engineering, Los Angeles, CA, Lewis, P., Hyle, P., Parrington, M., Clark, E., Boehm, B., Abts, C. and Manners, B., "Lessons Learned in Developing Commercial-Off-The-Shelf (COTS) Intensive Software Systems," FAA Software Engineering Resource Center, Atlantic City, NJ, October Albert, C. and Morris, E., "Commercial Item Acquisition: Considerations and Lessons Learned," CMU Software Engineering Institute, Pittsburgh, PA, June Abts, C., Boehm, B. and Bailey Clark, B., COCOTS: a COTS software integration cost model, Proceedings ESCOM-SCOPE 2000 Conference, Munich, Germany, Boehm, B., Abts, C., Brown, A., Chulani, S., Clark, B., Horowitz, E., Madachy, R., Reifer, D. and Steece, B.; Software Cost Estimation with COCOMO II, Prentice Hall PTR, Upper Saddle River, NJ, July 2000.

8 7. Abts, C., " A Perspective on the Economic Life Span of COTS-based Software Systems: the COTS-LIMO Model," Proceedings of the COTS Software Systems Workshop held in conjunction with ICSE 2000, Limerick, Ireland, May Abts, C., " COTS-Based Systems (CBS) Functional Density A Heuristic for Better CBS Design," Proceedings of the First International Conference on COTS-Based Software Systems, Orlando, FL, February Summary Notes, US Federal Aviation Administration COTS Operations & Maintenance Issues Workshops, May 10-11, 2000, Ocean City, New Jersey and July 19-20, 2000, Herndon, Virginia.