Synthesis of Existing Cost Models to Meet System of Systems Needs

Transcription

1 Paper #128 Synthesis of Existing Cost Models to Meet System of Systems Needs Jo Ann Lane University of Southern California Center for Software Engineering 941 W. 37th Place, SAL Room 328 Los Angeles, CA Dr. Barry Boehm University of Southern California Center for Software Engineering 941 W. 37th Place, SAL Room 328 Los Angeles, CA Abstract Today s need for more complex, more capable systems in a short timeframe is leading more organizations towards the integration of existing systems into network-centric, knowledge-based system-of-systems (SoS). Software and system cost model tools to date have focused on the software and system development activities of a single software system, but none to date adequately estimate the integration of multiple systems into an SoS. This paper presents an overview of the activities that must be included in an SoS cost model and describes an approach for estimating SoS effort using the Constructive Cost Model (COCOMO) suite of estimation tools to estimate SoS Lead System Integrator (LSI) effort as well as the total SoS development effort. Introduction Today s need for more complex, more capable systems in a short timeframe is leading more organizations towards the integration of existing systems into network-centric, knowledge-based system-of-systems (SoS). Software and system cost model tools to date have focused on the software and system development activities of a single system, but none to date adequately estimate the Lead System Integrator (LSI) effort associated with architecting the SoS framework and the acquisition, integration, and test of SoS components for that framework. Currently, the University of Southern California (USC) Center for Software Engineering (CSE) is working on such a cost model, the Constructive SoS Integration Cost Model (COSOSIMO). The COSOSIMO model has evolved over the last two years based on feedback obtained from USC CSE affiliates in industry and academia with respect to scope, cost drivers, and scale factors. Recent developments indicate that this effort might be sufficiently estimated by using a special SoS calibration or tailoring of the USC CSE Constructive System Engineering Cost Model (COSYSMO) (Boehm et al. 2005) integrated with an SoS Program Management cost model to estimate the effort associated with the LSI program management. This paper presents a functional overview of the initial COSOSIMO cost model and a description of the LSI activities to be estimated in terms of the overall SoS program Work Breakdown Structure (WBS). It then describes how the LSI activities split into management and

2 technical categories, how the current COSOSIMO size drivers and scale factors map to the proposed underlying sub-model parameters, and the associated implications for future COSOSIMO calibration and validation. Finally, this paper presents a case study showing how the various COSOSIMO sub-models and other COCOMO suite tools can used to create effort estimates to support the various LSI estimation needs as well as provide a total estimate for SoS systems engineering and development. Background What is So Different About SoS Effort and the Estimation of that Effort? We are seeing a growing trend in industry and DoD to quickly incorporate new technologies and expand the capabilities of legacy systems by integrating them with other legacy systems, Commercial-Offthe-Shelf (COTS) products, and new systems. With this development approach, we see new activities being performed to define the new architecture, identify sources to either supply or develop the required components, and then to integrate and test these high level components. Along with this system-of-systems (SoS) development approach, we have seen a new role in the development process evolve to perform these activities: that of the LSI. Today, there are fairly mature tools to support the estimation of the effort and schedule associated with the lower-level SoS component systems. For software development activities, there are the COCOMO II, Cost Xpert, Costar, PRICE S, SLIM, and SEER-SEM cost models. At the single system level, there is COSYSMO to estimate the system engineering effort and PRICE H and SEER-H to estimate hardware development costs. For COTS implementation and integration, there is Constructive COTS cost model (COCOTS) to estimate the effort associated with the assessment, tailoring, and glue-code implementation of COTS software products. (Boehm et al. 2005) However, none of these models includes LSI activities such as the definition of the SoS architecture, the solicitation and procurement process for the SoS components, and the integration of the SoS components into the SoS framework. COSOSIMO is a parametric model to compute just this effort. It is designed to estimate the up-front LSI effort associated with SoS abstraction, architecting, source selection, systems acquisition, and supplier and vendor oversight during development, as well as the effort associated with the later activities of integration, test, and change management. The goal is to support activities for estimating the LSI effort in a way that allows users to develop initial estimates and then conduct tradeoffs based on architecture and development process alternatives. With the addition of COSOSIMO to the COCOMO suite of estimation tools, there will be more complete coverage of the development activities associated with implementing a system-ofsystems. What is a System of Systems? Key to developing a cost model such as COSOSIMO is understanding what a system-of-systems is. Early literature research (Jamshidi 2005) and workshops with industry affiliates (Lam et al. 2004) show that the term system-of-systems means many things to many different people and organizations. In the business domain, it is the enterprise-wide and cross-enterprise integration and sharing of core business information across functional and geographical areas. In the military domain, it is a dynamic computing and communications infrastructure to support operations in a constantly changing, sometimes adversarial, environment. For some, an SoS may be a multi-system architecture that is planned up-front by an LSI. For others, an SoS is an architecture that evolves over time, often driven by emergent organization needs, new technologies appearing on the horizon, and available budget and schedule. The evolutionary SoS architecture is more of a network architecture that grows

3 with needs and available resources. Figure 1 shows a sample SoS that links together resources to support metropolitan crisis management activities. In any of these cases, users and nodes in the SoS network may be either fixed or mobile. Communications may be either Net - Centric SoS Figure 1. Sample SoS. point-to-point or broadcast. Networks may tie together other networks as well as nodes and users. SoS component systems typically come and go over time. These component systems can operate both within the SoS framework and independent of the SoS framework Andrew Sage and Christopher Cuppan best summed it up when they state that SoSs typically contain a majority of the following characteristics: operational independence of individual systems, managerial independence of individual systems, geographic distribution of SoS component systems, emergent SoS behaviour not contained in any one component system, and evolutionary development over time, with functionality continually being added, deleted, and modified. (Sage et al. 2001) For the purposes of cost modeling, it became clear early on that a single cost model could not support the variety of SoSs. As a result, efforts were initiated to better define the set of SoSs that would be supported by COSOSIMO (Lane et al. 2005). The key characteristics that identify the set of SoSs well-suited for cost modeling are: Strategically-oriented stakeholders interested in tradeoffs and costs Long-range architectural vision for SoS LSI selected to define overall architecture and manage development and integration System component independence. What is a Lead System Integrator? After defining the set of SoSs to be supported by COSOSIMO, the focus turned to defining an LSI in the SoS environment. In order to answer questions such as how much effort and how long, it was important to first understand the types of activities performed in SoS architecture development and integration and how these vary across different SoS implementations. Sources reviewed included LSI statements of work (or summaries of statements of work) that could be accessed on the web for new or currently ongoing LSI efforts. As a result, the COSOSIMO team came up with the following description (Lane 2005b): An LSI is an organization (or set of organizations) selected to oversee the definition, development and integration of an SoS. Typical LSI activities include: - Concurrent engineering of requirements, architecture, and plans - Identification and evaluation of technologies to be integrated - Source selection of vendors and suppliers - Management and coordination of supplier activities - Validation and feasibility assessment of SoS architecture - Continual integration and test SoS-level capabilities

4 - SoS-level implementation planning, preparation, and execution - On-going change management at the SoS level and across the SoS-related Integrated Product Teams (IPTs) to support the evolution of requirements, interfaces and technology. Typically LSIs do not develop system components to be integrated. The one possible exception is the development of the SoS infrastructure into which the system components will be integrated. What has become clear during research conducted to date on LSI activities it that these are activities that have typically been performed by government agencies (or their selected support contractors) or line management in the commercial world and the associated effort and costs have not been tracked closely. In addition, the typical system or software size drivers used in related cost estimation models do not always determine the associated effort for many of these activities. And finally, politics between SoS stakeholders, complex decision making processes with many required approvers, and competition between various suppliers and vendors can significantly impact this effort. Size Drivers Scale Factors COSOSIMO Estimation Approach The SoS cost estimation model, shown in Figure 2, is based on SoS and LSI key discriminators and characteristics that have a significant impact on effort and can be reasonably be estimated or known early in the inception and conceptualization phases. Therefore, the cost and size drivers are based on the SoS high-level architecture characteristics (product), processes used for design and integration/test (process), and LSI experience and capabilities (personnel). COSOSIMO Parameters. Calibration Figure 2. COSOSIMO Model Structure. SoS Definition and Integration Effort The selected size drivers focus on quantifiable attributes associated with the SoS interface protocols and the political complexities of negotiating agreements with other organizations to support the required system component changes needed to enable the SoS protocols (Lane 2005a). They are: Number of SoS Interface Protocols: The number of distinct interface protocols to be provided by the SoS framework. Number of Independent System Component Organizations: The number of organizations that are providing system components that will operate within the SoS framework. Each of these size inputs is weighted with respect to expected complexity and volatility. The interface protocol complexity is determined by factors such as number of protocol layers, desired security, degree of standardization, and whether this is a new protocol being implemented for the first time or it is an existing protocol. The independent system component organization complexity is determined by the expected level of cooperation between organizations and the competing needs of the SoS versus the needs and schedules of the independent component system provider or sponsor. The early-design scale factors, shown in Table 1, have been selected to capture the effects of key development processes and business/political decisions (Lane 2005a). The goal was to

5 include key influences with as few parameters as possible. Therefore, some of the COSOSIMO scale factors are really multi-attribute parameters that include several related influences. This allows the user to subjectively weight related influences based on the characteristics and dynamics of the SoS of interest. Scale Factor SoS Architecture Maturity Cost/Schedule Compression Integration Risk Resolution Component System Maturity and Stability Component System Readiness Integration Team Capability Maturity of the Integration Processes Table 1. COSOSIMO Scale Factors. Description A parameter that represents the level of maturity of the SoS architecture. It includes the level of detail of the interface protocols and the level of understanding of the performance of the protocols in the SoS framework. The extent of business or political pressures to reduce cost and schedule. A multi-attribute parameter that represents the number of major integration risk items, the maturity of risk management and mitigation plans, compatibility of schedules and budgets, expert availability, tool support, and level of uncertainty in integration risk areas. A multi-attribute parameter that indicates the average maturity level of the system components (number of new component systems versus number of component systems currently operational in other environments), overall compatibility of the system components with each other and the SoS interface protocols, the number of major component system changes being implemented in parallel with the SoS framework changes, and the anticipated change in the component systems during SoS integration activities. Indicates the readiness of component systems for integration. The user evaluates the average level of verification and validation that has/will be performed prior to integration and the level of subsystem integration activities that will be performed prior to integration into the SoS integration lab. Represents the anticipated level of integration team cooperation and cohesion, integration personnel capability and continuity, and integration personnel experience with the application, language, integration tools, and integration platform(s). A parameter that rates the maturity level and completeness of an integration team s integration processes, plans, and the SoS integration labs/test beds. How Does COSOSIMO Compare to the CSE System Engineering Cost Model? During the development of COSOSIMO, CSE industry affiliates have questioned whether COSOSIMO is really all that different than the system engineering cost model, COSYSMO. This is due in part to the considerable size driver and scale factor overlap between the two cost estimation models. In addition, many thought that the LSI effort associated with an SoS development was just a different system of interest as defined in (ISO 2002). So, in July 2005, several CSE industry affiliates participated in a survey to further investigate this view and to better understand the key activities performed on SoS LSI programs and the size drivers and scale factors related to both traditional system engineering and SoS LSI programs (Lane et al. 2005). The analysis of the survey responses indicates that: a. The system engineering activities identified in (Electronic Industries Alliance 1999) describe both the systems engineering and SoS LSI activities. However, the total SoS LSI effort spent on each of the activities is thought to be more than in the typical systems engineering project and the distribution of that effort is different than the systems engineering profile. (An attempt was made to quantify these differences, but very little data was provided on the survey forms.)

6 b. At first glance, most of the COSYSMO size drivers and scale factors are generally thought to be applicable to the effort associated with LSI technical activities. However, many of the definitions change in the SoS context. For example, in the SoS environment, Architecture Maturity is used to evaluate the maturity of the SoS architecture framework whereas Architecture Understanding evaluates the strength of understanding of the architecture at the system level. Maturity is used to convey that in the SoS environment, it is not sufficient to understand the architecture, rather testing, simulation, and modeling should be conducted to ensure the scalability and performance desired in the much more complex SoS environment. One of the consequences of an immature architecture is excessive rework later when SoS-level architecture changes are required and many of these changes must be propagated to supplier-vendors via contract modifications. Next, some scale factors in addition to those in the current version of COSYSMO are required to capture characteristics somewhat unique to the SoS development approaches. An example is Component System Maturity and Stability to evaluate the overall maturity and stability of the independently developed and managed SoS components. Finally, an SoS-unique calibration would be required to capture the effects of program aspects more unique to the very large LSI efforts. c. The COSYSMO model size drivers and scale factors are focused on technical effort and are not sufficient to estimate the management effort associated with LSI programs. Therefore, if a COSYSMO-like model is used to estimate the LSI technical effort, an additional program management cost model will be required to estimate the corresponding LSI management effort. This approach is attractive in that a program management cost model would be applicable to more than just the SoS LSI management activities. There are probably no significant cost savings to using COSYSMO to estimate the SoS LSI technical effort since COSYSMO would need to be modified, actual SoS LSI data collected and analyzed, and a separate calibration performed. (These are key steps in the COCOMO model development methodology as described in (Boehm et al. 2005)). In addition, there is the cost of developing the program management cost model. The benefit to the two-sub-model approach, as mentioned above, is that this effort will provide a separate program management cost model that can be used in conjunction with other COCOMO suite cost estimation models. The approach selected (either a single SoS cost model or the combined use of COSYSMO and a program management model) will depend on the data that can be collected from SoS programs. If LSIs can provide data that separates program management effort from technical effort, two sub-models can be developed. However, analysis of current LSI cost accounting systems indicates that this may be difficult to do. How to Estimate Total SoS Development Effort Using the COCOMO Suite of Tools: The goal of the COCOMO suite of cost estimation tools is support the estimation of large, complex systems engineering and software development activities. When initially analyzing the SoS development process, it became clear that many pieces of the SoS development could be estimated using existing cost models, as illustrated in Figure 3. During this analysis, some other key things became apparent about the SoS development processes: The upfront SoS architecture must be adequately defined in order to identify the desired system components and the interfaces necessary to support communications between the

7 Level 0 SOS Level 1 S 1 S 2 S m Level 2 S 11 S 12 S 1n S 21 S 22 S 2n S m1 S m2 S mn Activity SoS Lead System Integrator Effort (SoS scoping, planning, requirements, architecting; source selection; teambuilding, re-architecting, feasibility assurance with selected suppliers; incremental acquisition management; SoS integration and test; transition planning, preparation, and execution; and continuous change, risk, and opportunity management) Development of SoS Software-Intensive Infrastructure and Integration Tools Levels Level 0, and other levels if lower level systems components are also SoSs Level 0 Cost Model COSOSIMO COCOMO II System Engineering for SoS Components Software Development for Software-Intensive Components COTS Assessment and Integration for COTS-based Components Levels 1-n Levels 1-n Levels 1-n COSYSMO COCOMO II COCOTS Figure 3. Architecture-based Cost Estimation. SoS components. With a stable, SoS infrastructure architecture, component modifications and new development can be done independently of modifications for other system components, component modification/development estimates can be made more accurately, component rework is minimized, and SoS integration and test activities can be better planned and managed. SoSs tend to be evolutionary and therefore, detailed, long-term estimates are not typically feasible. What is more typical is that the over-arching architecture can be defined and developed along with the first several increments of the SoS. SoS development tends to be more schedule or cost driven, with stakeholders wanting to know what can be done in the next year or two with a given budget, and then decide how they want to evolve the SoS next. The future increments are often determined by new technology development, some of which is driven by SoS needs, some of which is developed independent of the SoS, but has applications within the SoS. Part of the SoS evolutionary process is technology refresh as COTS products evolve, network technology evolves, and unanticipated technology becomes available. One of the key SoS planning tools that supports the estimation process is the SoS WBS (Wang et al. 2006). The WBS identifies the SoS LSI activities described above as well as captures information about system components to be integrated into the SoS, the details of the near-term increments, and high-level information about longer-term increments. It also provides information about how dynamic and responsive to change the SoS development organization is. Recent work in analyzing SoS organizational structures (Boehm 2005) has shown that many are

8 adapting to the complexity of the SoS environment by integrating agile teams with more traditional plan-driven (PD) teams and continuous verification and validation (V&V) teams. The agile teams respond to the changing environment and define short, stable increments for development. The plan-driven teams implement capabilities in accordance with the stable increment definitions. The continuous V&V teams support the integration and test of the plandriven increments. The details (or lack of details) in the WBS can often indicate the maturity of the plans, how quickly the development organizations can respond to and manage change, and the corresponding accuracy of associated estimates. Figure 4 shows how the total SoS development effort can be viewed and estimated with respect to the COCOMO suite of estimation tools. Note that in the absence of a calibrated COSOSIMO model, COSYSMO can be used to estimate the LSI technical effort and nonparametric methods (e.g., percentage of total effort, activity-based costing) can be used to estimate the other LSI program management activities. Figure 4. SoS Cost Estimation Across Life Cycle Phases and Increments. The initial up-front architecting of the SoS system in the SoS Inception phase, resulting in a Life Cycle Objectives (LCO) review that ensures that there are feasible options for the desired SoS, can be estimated using a COSYSMO-like model with parameters selected to best describe the SoS product, process, and LSI team characteristics. Once the total estimate is computed, it must be adjusted to reflect just the Inception effort. (The current COSYSMO model analysis suggests allocating 7% of the total effort for Inception, 16% for Elaboration, 35% for Development, 28% for Test and Evaluation, and 14% for Transition.)

9 In the next phase, the SoS Elaboration phase, the LSI team must identify the specific system components to be integrated into the SoS, develop Requests for Proposals to solicit responses from prospective vendors, select vendors, adjust the architecture to be consistent with the selected vendors, and then conduct a Life Cycle Architecture (LCA) review at the SoS level to show the feasibility of the selected approach. A key part of this effort is the evaluation of the supplier and vendor proposals, looking at the completeness and feasibility of their approaches, and identifying potential risks or rework that might result from their approaches. This effort must be estimated using a COSOSIMO-like model since many of these activities are not typically part of a more traditional systems engineering effort. As the supplier and vendor contracts are put in place, work in the first increment begins. The suppliers/vendors begin working to the plans that the LSI teams developed in the Elaboration phase, using their plan-driven teams. This activity begins with LCA reviews at the supplier level to ensure the feasibility of the supplier s approach and to identify any additional risks to be tracked during the development. During the early SoS increments, the LSI may also have supplier teams that are responsible for developing the SoS infrastructure. These development efforts are estimated for each system component using a combination of COSYSMO for the component system engineering effort and either a COCOMO II-like cost model or, for more rapid development processes, a CORADMO-like cost model for the associated software development effort. At the same time the suppliers and vendors are working on Increment 1, the LSI agile teams are assessing potential sources of change, rebaselining requirements for the next increment, and negotiating those changes with the suppliers and vendors. In addition, the LSI V&V team is continually monitoring, verifying, and validating Increment 1, resulting in an Initial Operational Capability (IOC) review for Increment 1. These LSI planning, adjustment, and V&V efforts are estimated using a COSOSIMO-like model. As one increment is completed, the plan-driven development teams begin work on the subsequent increment, n, that has been defined just-in-time by the agile teams. And the agile teams continue their forward-looking work on increment n+1. By putting these pieces together for the known SoS increments, it is possible to develop a fairly accurate estimate of the total SoS development for the defined increments. Conclusions The goal of this paper was to show how existing cost models could be synthesized to support the estimation of SoS development efforts. It has been clear for a while that the current systems engineering and software development cost models do not completely address all of the effort in today s SoS arena. However, by viewing the SoS development as the planning and coordination of many projects (the SoS architecture and the development/modifications to system components), many of today s cost models can be used to estimate significant pieces of this effort. The key to an accurate estimate is to have a sufficiently defined architecture that allows system components to be developed or modified to operate in the SoS environment with few dependencies on other SoS development efforts. This process reduces risk and rework and allows the system component pieces to be estimated as if they were fairly independent projects using existing systems engineering and software development cost models. The final major source of effort, that of the LSI, still needs to be estimated using a new cost model such as COSOSIMO. Whether this is implemented as a single cost model or a set of submodels remains to be seen. Efforts are currently underway to collect and analyze LSI SoS data.

10 References Boehm, B., Valerdi, R., Lane, J., and Brown, W., COCOMO Suite Methodology and Evolution, CrossTalk, April Boehm, B., Some Future Trends and Implications for Systems and Software Engineering Processes, USC-CSE-TR , (also accepted for publication in Systems Engineering, 2006) Electronic Industries Alliance, EIA Standard 632: Processes for Engineering a System, January International Organization for Standardization (ISO), Systems Engineering System Life Cycle Processes, ISO/IEC 15288, Jamshidi, M., System-of-Systems Engineering - a Definition, IEEE SMC 2005, Hawaii, October Lam, A. and Lane, J., COSOSIMO Workshop Minutes, Proceedings of 19th Forum on COCOMO and Software Cost Modeling, USC CSE, October 26, Lane, J., Factors Influencing System-of-Systems Architecting and Integration Costs, Proceedings of Conference on System Engineering Research, March Lane, J., System of Systems (SoS) Processes, Proceedings of USC CSE Annual Research Review, March Lane, J. and R. Valerdi, Synthesizing SoS Concepts for Use in Cost Estimation, Proceedings of IEEE SMC Conference, October Sage, A. and C. Cuppan. "On the Systems Engineering and Management of Systems of Systems and Federations of Systems." Information, Knowledge, and Systems Management 2 (2001): Wang, G., R. Valerdi, J. Lane, and B. Boehm, Towards a Work Breakdown Structure for Net Centric System of Systems Engineering and Management, INCOSE Symposium, Biography Jo Ann Lane is currently a research assistant supporting software engineering and system of systems research activities at the University of Southern California Center for Software Engineering. In this capacity, she is currently working on a cost model to estimate the effort associated with system of systems architecture definition and integration. Prior to this, she was a key technical member of Science Applications International Corporation s (SAIC) Software and Systems Integration Group. She has over 28 years of experience in the development of softwareintensive systems. Barry Boehm, Ph.D., is the TRW professor of software engineering and director of the Center for Software Engineering at the University of Southern California. He was previously in technical and management positions at General Dynamics, Rand Corp., TRW, the Defense Advanced Research Projects Agency, where he managed the acquisition of more than $1 billion worth of advanced information technology systems. Dr. Boehm originated the spiral model, the Constructive Cost Model, and the stakeholder win-win approach to software management and requirements negotiation.