System Cost Modeling Using Proxy Estimation and COSYSMO October 21, 2014 Reggie Cole Lockheed Martin Senior Fellow reggie.cole@lmco.com Kevin Woodward Lockheed Martin Fellow kevin.woodward@lmco.com 1
We Have a Problem That COSYSMO Can Address BBP 3.0 Mandates: Should-Cost Design-to-Cost Design for Affordability The Tool Gap But We Have a Tool Gap Early-Stage Cost Modeling System Cost Modeling COSYSMO Can Be Extended to Cover the System Cost Modeling Gap 2
Background on Previous Work Between 2007 and 2012 We realized that COSYSMO could provide insight into changing SE effort not just an initial estimate Which in turn allowed us to understand how the overall system scope was changing in other words it was a proxy We found a way to change the proxy into system cost estimate COCOMO Forum 2012 We presented the initial approach, Proxy Estimation Costing for Systems (PECS) We started using what we called PECS for design-to-cost analysis and analysis of alternatives COCOMO Forum 2013 We conducted a workshop on the approach and made the decision to consider integrating it with the next revision of COSYSMO Over the Past Year We have continued to use the approach, starting planning for broader validation and started discussing how the approach would be integrated with COSYSMO 3
SE Effort is a Proxy Measure of Overall System Size and Complexity System Engineering Effort is a Proxy Measure for System Cost There is strong evidence for the link between systems engineering effort and program cost dating back to a NASA study in the 1980s The optimal relationship between systems engineering effort and overall program cost is 10% - 15% Industry has long used a parametric relationship between software cost and systems engineering cost for software-intensive systems Systems engineering effort can be an effective proxy measure for overall system cost But We Can Convert the Systems Engineering Effort Into a System Cost Estimate Develop a bias function the that turns the biased proxy estimator into an unbiased true estimator Use the established relationship between SE effort and total system effort to partially de-bias the proxy estimate Add in additional parameters to complete the de-biasing process 4
Actual/Planned Cost Putting It All Together Size Drivers (Problem Space) Customer Requirements System Interfaces Major Algorithms Operational Scenarios Complexity Drivers (Problem/Solution) Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Documentation Needs Installations/Platform Diversity Levels of Recursion in the Design Stakeholder Team Cohesion Personnel/Team Capability Personnel Experience/Continuity Process Capability Multisite Coordination Tool Support Reuse Factors (Solution Space) New Modified Deleted Adopted Managed SE Effort is an estimator for total system cost but it is a biased estimator Estimator De-Biasing Monte Carlo Analysis of System Cost 3.0 2.6 2.2 1.8 1.4 1.0 0.6 Estimator Bias Function is Based on the Well- Established Relationship Between SE Effort and Overall Program Effort 0% 4% 8% 12% 16% 20% 24% 28% Where : Cal Conv SE Effort = SE Quality * SE Cost/Actual Cost Cal Labor CostSystem Effort SE StochasticAdders Cost FConv F F Effort Rate :COSYSMO Calibration Factor (Deterministic) : Factor for Converting SE Effort tototalprogram Effort (Stochastic) SE Labor :SE Effort ComputedUsing COSYSMO (Stochastic) StochasticAdders : Labor Rate (Stochastic) Cost F Deter min isticadder s Deter min isticadder s : Additional Factors (e.g., travel, material, etc.) (Stochastic) Cost Rate : Additional Factors (G & A,ODC, MR,Fee, etc.) (Deterministic) Cost Three different COSYMO scenarios optimistic, expected & pessimistic provide the basis for the Monte Carlo analysis of system cost 5
Adjusting Our View of COSYSMO Parameters Our view of the size drivers can be preserved their context doesn t change under COSYSMO extension Our view of reuse requires the most extreme adjustment we are not just talking about systems engineering artifacts Our view of the cost drivers needs to be broadened to include all aspects of the system, not just systems engineering 6
Adjusting Our View of Cost Drivers Cost Drivers Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Level of Documentation Required Diversity of Installed Platforms Level of Design Recursion Stakeholder Team Cohesion Personnel / Team Capability Personnel Experience / Continuity Process Capability Multisite Coordination Level of Tool Support Precedented systems and unprecedented systems are fundamentally different System modification and reuse have a significant effect on some cost drivers Need to consider the entire team including subcontractors Need to consider the all processes and tools 7
Adjusting Our View of Reuse Factors New Elements These are elements that need to be engineered and developed. Just reusing systems engineering artifacts is not sufficient. Reuse Factors New Elements Modified Elements Deleted Elements Adopted Elements Managed Elements Modified Elements These are elements that offer some form of reuse. Enhanced COTS or reusable components that need modification fall into this category. Adopted Elements These are elements that offer significant reuse with minimal modification and do not require full retesting. COTS typically falls into this category. For Requirements Think Functional Components For Interfaces Think Connection Points For Algorithms Think Functional Components For Scenarios Think Implications to Both Managed Elements These are elements that are already in the system and require minimal regression testing. A previously deployed element falls into this category. 8
The Bias Function Cal Labor CostSystem Effort SE StochasticAdders Cost FConv Where : F F Cal Conv Effort Rate : COSYSM O Calibration Factor (Deterministic) : Factor for Converting SE Effort tototal Program Effort (Stochastic) SE Labor :SE Effort Computed Using COSYSM O (Stochastic) StochasticAdders : Labor Rate (Stochastic) Cost F Deter min isticadder s : Additional Cost Rate : Additional Deter min isticadder s Factors (e.g., travel, material, etc.) (Stochastic) Variable Type Description Cost Factors (G & A,ODC, MR,Fee, etc.) (Deterministic) COSYSMO Calibration Factor Deterministic Scalar Value Organization-specific calibration factor Effort Conversion Factor Triangular Distributed Random Variable Three-point estimate of factor to convert SE effort to total program effort (nominally 0.08, 0.12 and 0.16) SE Effort Triangular Distributed Random Variable Three-point estimate for SE effort, generated using COSYSMO Labor Rate Triangular Distributed Random Variable Three-point estimate for composite labor rate Material Costs Triangular Distributed Random Variable Three-point estimate for material costs Travel Costs Triangular Distributed Random Variable Three-point estimate for travel costs We are going to discuss each of these factors! 9
SE Effort (Effort SE ) COSYSMO provides a Proxy Estimate of the system cost. We will not try to de-bias it right now.that is the next step. Size Drivers (Problem Space) Customer Requirements System Interfaces Major Algorithms Operational Scenarios Complexity Drivers (Problem/Solution) Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Documentation Needs Installations/Platform Diversity Levels of Recursion in the Design Stakeholder Team Cohesion Personnel/Team Capability Personnel Experience/Continuity Process Capability Multisite Coordination Tool Support Since we want to perform Monte Carlo simulation of costs, we would like to generate a distribution of the proxy costs. Three different COSYMO scenarios optimistic, expected & pessimistic provide the basis for a sampling distribution. Optimistic Expected Pessimistic Reuse Factors (Solution Space) New Modified Deleted Adopted Managed COSYSMO Estimate of Hours Becomes the Parameters for Either Triangular or Beta Pert Distribution Triangular Distribution Beta Pert Distribution 10
Actual/Planned Cost Program Overrun Effort Conversion Factor (F Conv ) This is probably the most important factor in the bias function! 3.0 200 180 160 140 120 100 80 60 40 20 Total Program Overrun 32 NASA Programs Definition $ Definition Percent = ---------------------------------- Target + Definition$ Actual + Definition$ Program Overrun = ---------------------------------- Target + Definition$ R 2 = 0.5206 0 0 5 10 15 20 Definition Percent of Total Estimate Studies provide some insight into what the value should be Heuristic approaches for determining SE costs for software intensive systems are also consistent with these studies All indications point to a range of 0.08 to 0.16 for F Conv And this range is consistent with all the data we ve collected to date for relatively healthy programs 2.6 2.2 And it provides the basis for our random variable parameters 1.8 1.4 1.0 0% 4% 8% 12% 16% 20% 24% 28% Pessimistic Expected Optimistic 0.08 0.12 0.16 0.6 SE Effort = SE Quality * SE Cost/Actual Cost Triangular Distribution Beta Pert Distribution 11
Other Stochastic Adder Factors Labor Costs Labor costs vary especially in early stages and needs to be treated as a random variable Any number of distributions would probably be OK Beta Pert would be a good default If hours are the item of interest rather than cost, this factor can be omitted Material Costs Material costs vary especially in early stages and needs to be treated as a random variable Any number of distributions would probably be OK Beta Pert would be a good default but there is at least one study that looks at using a normal distribution If hours are the item of interest rather than cost, this factor can be omitted Travel Costs Travel costs vary especially in early stages and needs to be treated as a random variable Any number of distributions would probably be OK Beta Pert would be a good default If hours are the item of interest rather than cost, this factor can be omitted Other Any number of other stochastic adders can be treated similarly 12
Other Deterministic Adders Some Factors Are Well Known To the Point They Can Be Considered Deterministic They are often set and apply across programs Examples include: G&A Costs Other Direct costs Management Reserve Fee 13
Consideration of Additional Parameters 2013 Workshop Identified the Need to Take a Closer Look at the Parameters The meaning, interpretation and scope of existing COSYSMO parameters COCOMO parameters that might be meaningful when COSYSMO is used in this way 14
Analysis of Additional Parameters COSYSMO COCOMO There seems to be a lot of system cost modeling relevancy in the COCOMO parameters There seems to be a lot of correlation between COSYSMO and COCOMO with respect to system cost modeling 15
The Key Use Cases Should-Cost Analysis Establishing a should-cost for which cost estimates from bidders can be evaluated Establishing a DTC target for performing DTC analysis Design-to-Cost Analysis Performing cost vs. capability trades as way to provide more affordable solutions Analysis of Alternatives Evaluating alternative solution strategies It s not just about the problem space the solution space can be evaluated too 16
Should-Cost Analysis Example Optimistic Expected Pessimistic Requirements Baseline Requirements Interfaces Algorithms Architecture Baseline Scenarios Use a Plug Number for Adders Anticipated Distribution of Labor Rates Anticipated Distribution of Material Costs Anticipated Travel Costs Anticipated Supplier Fees Bias Function Risky Range Target Cost Target Reserve 17
DTC Example Cost Analysis Requirements Baseline Requirements Interfaces Optimistic Expected Pessimistic Architecture Baseline Algorithms Scenarios Three COSYSMO Scenarios for Each Capability Use a Common Number for Adders Anticipated Distribution of Labor Rates Anticipated Distribution of Material Costs Anticipated Travel Costs Anticipated Supplier Fees Bias Function A Cost Curve is Produced for Each Capability Take the 80% Confidence Cost as the Capability Cost 18
DTC Example Capability Trade-Off Capabilities Sorted By Cost Colors Map Mission Utility Priorities RED = High YELLOW = Medium Green = Low Capabilities Sorted By Mission Utility (Cost for Priority 1 Capabilities Only) (Capabilities at DTC Target) (Cost for Priority 1&2 Capabilities) (Cost for All Capabilities) 19
Analysis of Alternatives Example 20-Year Old C2 System The system was unprecedented at the time Twenty years later, a replacement system is needed due to obsolescence and needed changes Alternative Solutions Full Replacement System Develop and deploy a new replacement system COTS/GOTS/NDI-Based Replacement System Use a combination existing non-obsolete components and COTS components Modify components as necessary to meet requirements 20
The Two Key Alternatives Newly Developed System Ground-Up Development of System Requirements Refinement, Architecture, Detailed Design Soup-to-Nuts But It is No Longer an Unprecedented System So Requirements/Architecture Understanding, etc. Are High Refurbished System Using Modified NDI Use of NDI is Maximized Additional Components Developed as Necessary to Meet Requirements Reuse Considerations Drive This Alternative 21
COSYSMO Size and Cost Drivers Size Drivers for the Two Alternatives Are Largely the Same Cost Drivers for the Two Alternatives Are Very Similar With a Few Notable Exceptions Cost Drivers Requirements Understanding Architecture Understanding Level of Service Requirements Migration Complexity Technology Risk Level of Documentation Required Diversity of Installed Platforms Level of Design Recursion Stakeholder Team Cohesion Personnel / Team Capability Personnel Experience / Continuity Process Capability Multisite Coordination Level of Tool Support System modification and reuse have an effect on some cost drivers 22
COSYSMO Reuse Factors Reuse Factors New Elements Modified Elements Deleted Elements Adopted Elements Managed Elements New Elements These are elements that need to be engineered and developed. Just reusing systems engineering artifacts is not sufficient. Modified Elements These are elements that offer some form of reuse. Enhanced COTS or reusable components that need modification fall into this category. Deleted Elements For the NDI Alternative, Some Elements May Need to Be Deleted/Retired Adopted Elements These are elements that offer significant reuse with minimal modification and do not require full retesting. COTS typically falls into this category. Most Elements Are New for the Development Alternative Most Elements Are Modified for the NDI Alternative Elements That Need to Be Retired Should Be Treated as Deleted Elements Elements That Are Wrapped Can Be Treated as Adopted for the NDI Alternative Managed Elements These are elements that are already in the system and require minimal regression testing. A previously deployed element falls into this category. Elements That Stay in Place Without Modification (or Wrapping) Can Be Treated as Managed 23
Cost Analysis Approach Requirements Baseline Requirements Interfaces Optimistic Expected Pessimistic Architecture Baseline Algorithms Scenarios Three COSYSMO Scenarios for Each Alternative Use a Common Number for Adders Anticipated Distribution of Labor Rates Anticipated Distribution of Material Costs Anticipated Travel Costs Anticipated Supplier Fees Bias Function A Cost Curve is Produced for Each Capability Take the 80% Confidence Cost as the Capability Cost 24
But This is Not Really COSYSMO Right? The Scope of COSYSMO is Systems Engineering Effort But the scope of this approach is total delivered system cost COSYSMO is a Fully Parametric Model But this is a hybrid model that consists of parametric modeling and Monte Carlo simulation This Approach is Not COSYSMO It uses COSYSMO to develop the proxy estimate in other words, COSYSMO serves as an input Eventually This Will Likely Become Part of COSYSMO Those discussions are ongoing 25
Summary COSYSMO provides the basis for estimation of systems engineering effort and a biased proxy estimator for overall system cost There is a well-established relationship between systems engineering effort and overall effort used to de-bias the COSYSMO-modeled effort This approach can improve system cost modeling but it is not a replacement for existing models This approach occupies an important niche fully parametric system cost modeling in the early stages of system definition This approach can serve as a powerful affordability analysis tool supporting should-cost, DTC and AoA 26
27