A Value-Based Orthogonal Framework for Improving Life-Cycle Affordability

Transcription

1 A Value-Based Orthogonal Framework for Improving Life-Cycle Affordability Barry Boehm, Jo Ann Lane, Sue Koolmanojwong NDIA Systems Engineering Conference October 25, 2012

2 Outline Affordability definitions, concepts, issues, strategies Addressing both costs and benefits Using life cycle present value Coping with uncertainty: incrementally; pro-actively Coping with multi-stakeholder value diversity Addressing tradeoffs with other -ilities An orthogonal framework for improving affordability costs Cost modeling and other insights Conclusions October 16, 2012 Copyright USC-CSSE 2

3 Affordability Definitions INCOSE: The balance of system performance, cost, and schedule constraints over the system life cycle, while satisfying mission needs in concert with strategic and organizational needs. MORS: Cost-effective capability (USD/ATL). NDIA: The practice of assuring program success through the balancing of system performance (KPPs), cost, and schedule constraints, while satisfying mission needs in concert with the long-range investment and force structure plans of the DoD. Webster: Keeping within your financial means. October 16, 2012 Copyright USC-CSSE 3

4 Affordability Concepts Coping with uncertainty: incrementally; pro-actively Coping with multi-stakeholder value diversity Life Cycle Benefits Improvement Primary Option Space Achievable Pareto Boundary Current Situation Increasing Cost-Benefit Improvement Isoquants Life Cycle Cost Improvement October 16, 2012 Copyright USC-CSSE 4

5 Multi-Stakeholder Value Diversity Bank of America Master Net 5 October 16, 2012 Copyright USC-

6 Outline Affordability definitions, concepts, issues, strategies Addressing both costs and benefits Using life cycle present value Coping with uncertainty: incrementally; pro-actively Coping with multi-stakeholder value diversity Addressing tradeoffs with other -ilities An orthogonal framework for improving affordability costs Cost modeling and other insights Conclusions October 16, 2012 Copyright USC-CSSE 6

7 Get the Best from People Affordability and Tradespace Framework Staffing, Incentivizing, Teambuilding Facilities, Support Services Kaizen (continuous improvement) Affordability Improvements and Tradeoffs 7 Make Tasks More Efficient Eliminate Tasks Eliminate Scrap, Rework Simplify Products (KISS) Reuse Components Reduce Operations, Support Costs Value- and Architecture-Based Tradeoffs and Balancing Copyright October 16, USC-CSSE 2012 Tools and Automation Work and Oversight Streamlining Collaboration Technology Lean and Agile Methods Task Automation Model-Based Product Generation Early Risk and Defect Elimination Evidence-Based Decision Gates Modularity Around Sources of Change Incremental, Evolutionary Development Value-Based, Agile Process Maturity Risk-Based Prototyping Value-Based Capability Prioritization Satisficing vs. Optimizing Performance Domain Engineering and Architecture Composable Components,Services, COTS Legacy System Repurposing Automate Operations Elements Design for Maintainability, Evolvability Streamline Supply Chain Anticipate, Prepare for Change

8 Costing Insights: COCOMO II Productivity Ranges Scale Factor Ranges: 10, 100, 1000 KSLOC Development Flexibility (FLEX) Team Cohesion (TEAM) Develop for Reuse (RUSE) Precedentedness (PREC) Architecture and Risk Resolution (RESL) Platform Experience (PEXP) Data Base Size (DATA) Required Development Schedule (SCED) Language and Tools Experience (LTEX) Process Maturity (PMAT) Storage Constraint (STOR) Use of Software Tools (TOOL) Platform Volatility (PVOL) Applications Experience (AEXP) Multi-Site Development (SITE) Documentation Match to Life Cycle Needs (DOCU) Required Software Reliability (RELY) Personnel Continuity (PCON) Time Constraint (TIME) Programmer Capability (PCAP) Analyst Capability (ACAP) Product Complexity (CPLX) Staffing Teambuilding Continuous Improvement Productivity Range October 16, 2012 Copyright 8 USC-CSSE

9 COSYSMO Sys Engr Cost Drivers Teambuilding Continuous Improvement Staffing October 16, 2012 Copyright USC-CSSE 9

10 Get the Best from People Tradespace and Affordability Framework Staffing, Incentivizing, Teambuilding Facilities, Support Services Kaizen (continuous improvement) Affordability Improvements and Tradeoffs 10 Make Tasks More Efficient Eliminate Tasks Eliminate Scrap, Rework Simplify Products (KISS) Reuse Components Reduce Operations, Support Costs Value- and Architecture-Based Tradeoffs and Balancing Copyright October 16, USC-CSSE 2012 Tools and Automation Work and Oversight Streamlining Collaboration Technology Lean and Agile Methods Task Automation Model-Based Product Generation Early Risk and Defect Elimination Evidence-Based Decision Gates Modularity Around Sources of Change Incremental, Evolutionary Development Value-Based, Agile Process Maturity Risk-Based Prototyping Value-Based Capability Prioritization Satisficing vs. Optimizing Performance Domain Engineering and Architecture Composable Components,Services, COTS Legacy System Repurposing Automate Operations Elements Design for Maintainability, Evolvability Streamline Supply Chain Anticipate, Prepare for Change

11 COCOMO II Productivity Ranges Scale Factor Ranges: 10, 100, 1000 KSLOC Development Flexibility (FLEX) Team Cohesion (TEAM) Develop for Reuse (RUSE) Precedentedness (PREC) Architecture and Risk Resolution (RESL) Platform Experience (PEXP) Data Base Size (DATA) Required Development Schedule (SCED) Language and Tools Experience (LTEX) Process Maturity (PMAT) Storage Constraint (STOR) Use of Software Tools (TOOL) Platform Volatility (PVOL) Applications Experience (AEXP) Multi-Site Development (SITE) Documentation Match to Life Cycle Needs (DOCU) Required Software Reliability (RELY) Personnel Continuity (PCON) Time Constraint (TIME) Programmer Capability (PCAP) Analyst Capability (ACAP) Product Complexity (CPLX) Tools Collaboration Technology Work Streamlining Productivity Range October 16, 2012 Copyright 11 USC-CSSE

12 Get the Best from People Tradespace and Affordability Framework Staffing, Incentivizing, Teambuilding Facilities, Support Services Kaizen (continuous improvement) Affordability Improvements and Tradeoffs 12 Make Tasks More Efficient Eliminate Tasks Eliminate Scrap, Rework Simplify Products (KISS) Reuse Components Reduce Operations, Support Costs Value- and Architecture-Based Tradeoffs and Balancing Copyright October 16, USC-CSSE 2012 Tools and Automation Work and Oversight Streamlining Collaboration Technology Lean and Agile Methods Task Automation Model-Based Product Generation Early Risk and Defect Elimination Evidence-Based Decision Gates Modularity Around Sources of Change Incremental, Evolutionary Development Value-Based, Agile Process Maturity Risk-Based Prototyping Value-Based Capability Prioritization Satisficing vs. Optimizing Performance Domain Engineering and Architecture Composable Components,Services, COTS Legacy System Repurposing Automate Operations Elements Design for Maintainability, Evolvability Streamline Supply Chain Anticipate, Prepare for Change

13 Trends in Software Expansion (Bernstein, 1997) MBSE:2010 Expansion Factor The ratio of machine lines of code to a source line of code Order of Magnitude Increase Every Twenty Years Machine Instructions Macro Assembler High Level Language Database Manager Regression Testing On-line Prototyping 4GL Subsecond Time Sharing Small Scale Reuse Object Oriented Programming Large Scale Reuse 13 October 16, 2012 Copyright USC-CSSE

14 Get the Best from People Tradespace and Affordability Framework Staffing, Incentivizing, Teambuilding Facilities, Support Services Kaizen (continuous improvement) Affordability Improvements and Tradeoffs 14 Make Tasks More Efficient Eliminate Tasks Eliminate Scrap, Rework Simplify Products (KISS) Reuse Components Reduce Operations, Support Costs Value- and Architecture-Based Tradeoffs and Balancing Copyright October 16, USC-CSSE 2012 Tools and Automation Work and Oversight Streamlining Collaboration Technology Lean and Agile Methods Task Automation Model-Based Product Generation Early Risk and Defect Elimination Evidence-Based Decision Gates Modularity Around Sources of Change Incremental, Evolutionary Development Value-Based, Agile Process Maturity Risk-Based Prototyping Value-Based Capability Prioritization Satisficing vs. Optimizing Performance Domain Engineering and Architecture Composable Components,Services, COTS Legacy System Repurposing Automate Operations Elements Design for Maintainability, Evolvability Streamline Supply Chain Anticipate, Prepare for Change

15 Get the Best from People Tradespace and Affordability Framework Staffing, Incentivizing, Teambuilding Facilities, Support Services Kaizen (continuous improvement) Affordability Improvements and Tradeoffs 15 Make Tasks More Efficient Eliminate Tasks Eliminate Scrap, Rework Simplify Products (KISS) Reuse Components Reduce Operations, Support Costs Value- and Architecture-Based Tradeoffs and Balancing Copyright October 16, USC-CSSE 2012 Tools and Automation Work and Oversight Streamlining Collaboration Technology Lean and Agile Methods Task Automation Model-Based Product Generation Early Risk and Defect Elimination Evidence-Based Decision Gates Modularity Around Sources of Change Incremental, Evolutionary Development Value-Based, Agile Process Maturity Risk-Based Prototyping Value-Based Capability Prioritization Satisficing vs. Optimizing Performance Domain Engineering and Architecture Composable Components,Services, COTS Legacy System Repurposing Automate Operations Elements Design for Maintainability, Evolvability Streamline Supply Chain Anticipate, Prepare for Change

16 Sequential Requirements-First Risks It s not a requirement if you can t afford it $100M $50M Required Architecture: Custom; many cache processors Original Budget Original Spec Original Architecture: Commercial DBMS After Prototyping Response Time (sec) 16 October 16, 2012 Copyright USC-CSSE

17 Get the Best from People Tradespace and Affordability Framework Staffing, Incentivizing, Teambuilding Facilities, Support Services Kaizen (continuous improvement) Affordability Improvements and Tradeoffs 17 Make Tasks More Efficient Eliminate Tasks Eliminate Scrap, Rework Simplify Products (KISS) Reuse Components Reduce Operations, Support Costs Value- and Architecture-Based Tradeoffs and Balancing Copyright October 16, USC-CSSE 2012 Tools and Automation Work and Oversight Streamlining Collaboration Technology Lean and Agile Methods Task Automation Model-Based Product Generation Early Risk and Defect Elimination Evidence-Based Decision Gates Modularity Around Sources of Change Incremental, Evolutionary Development Value-Based, Agile Process Maturity Risk-Based Prototyping Value-Based Capability Prioritization Satisficing vs. Optimizing Performance Domain Engineering and Architecture Composable Components,Services, COTS Legacy System Repurposing Automate Operations Elements Design for Maintainability, Evolvability Streamline Supply Chain Anticipate, Prepare for Change

18 The HMMWV 300,000 Produced, 22 Fielded Versions Initial draft requirements in 1979, Initial delivery in 1984 Mattracks or wheels Bolt on armor required upgraded suspension, engine, and steering Upgrades: Increased cab space Increased payload capacity Strengthened frame Additional armor and cupola raise the CG and increase rollovers Upper deck space is always at a premium Imbalance in cupola required motorized drive Suspension and steering for CG shift Base cab & flatbed with mission modules 9

19 Get the Best from People Tradespace and Affordability Framework Staffing, Incentivizing, Teambuilding Facilities, Support Services Kaizen (continuous improvement) Affordability Improvements and Tradeoffs 19 Make Tasks More Efficient Eliminate Tasks Eliminate Scrap, Rework Simplify Products (KISS) Reuse Components Reduce Operations, Support Costs Value- and Architecture-Based Tradeoffs and Balancing Copyright October 16, USC-CSSE 2012 Tools and Automation Work and Oversight Streamlining Collaboration Technology Lean and Agile Methods Task Automation Model-Based Product Generation Early Risk and Defect Elimination Evidence-Based Decision Gates Modularity Around Sources of Change Incremental, Evolutionary Development Value-Based, Agile Process Maturity Risk-Based Prototyping Value-Based Capability Prioritization Satisficing vs. Optimizing Performance Domain Engineering and Architecture Composable Components,Services, COTS Legacy System Repurposing Automate Operations Elements Design for Maintainability, Evolvability Streamline Supply Chain Anticipate, Prepare for Change

20 Post-Acquisition Costs Dominate (%O&M) Hardware [Redman 2008] 12% -- Missiles (average) 60% -- Ships (average) 78% -- Aircraft (F-16) 84% -- Ground vehicles (Bradley) Software [Koskinen 2010] 75-90% -- Business, Command-Control 50-80% -- Complex platforms as above 10-30% -- Simple embedded software Apply lack-of-flexibility factor to O&M component October 16, 2012 Copyright USC-CSSE 20

21 Get the Best from People Tradespace and Affordability Framework Staffing, Incentivizing, Teambuilding Facilities, Support Services Kaizen (continuous improvement) Affordability Improvements and Tradeoffs 21 Make Tasks More Efficient Eliminate Tasks Eliminate Scrap, Rework Simplify Products (KISS) Reuse Components Reduce Operations, Support Costs Value- and Architecture-Based Tradeoffs and Balancing Copyright October 16, USC-CSSE 2012 Tools and Automation Work and Oversight Streamlining Collaboration Technology Lean and Agile Methods Task Automation Model-Based Product Generation Early Risk and Defect Elimination Evidence-Based Decision Gates Modularity Around Sources of Change Incremental, Evolutionary Development Value-Based, Agile Process Maturity Risk-Based Prototyping Value-Based Capability Prioritization Satisficing vs. Optimizing Performance Domain Engineering and Architecture Composable Components,Services, COTS Legacy System Repurposing Automate Operations Elements Design for Maintainability, Evolvability Streamline Supply Chain Anticipate, Prepare for Change

22 Cost ($M) University of Southern California Tradeoffs Among Cost, Schedule, and Reliability, and Functionality: COCOMO II For 100-KSLOC set of features Can pick all three with 77-KSLOC set of features Development Time (Months) (RELY, MTBF (hours)) (VL, 1) (L, 10) (N, 300) (H, 10K) (VH, 300K) -- Cost/Schedule/RELY: pick any two points 22 October 16, 2012 Copyright USC-CSSE

23 A Value-Priority Tradeoff Equalizer October 16, 2012 Copyright USC-CSSE 23

24 Conclusions Affordability increasingly competition-critical Need to balance cost, schedule, performance, functionality Orthogonal framework helps tailor improvements Getting the best from people Making tasks more efficient Eliminating tasks Eliminating scrap and rework Simplifying products Reusing assets Reducing operations and support costs Value- and architecture-based tradeoffs and balancing No one-size-fits-all solution October 16, 2012 Copyright USC-CSSE 24

25 Backup Charts October 16, 2012 Copyright USC-CSSE 25

26 Agile and Plan-Driven Home Grounds: Five Critical Decision Factors Size, Criticality, Dynamism, Personnel, Culture Personnel (% Level 1B) (% Level 2&3) Criticality (Loss due to impact of defects) Dynamism (% Requirements - change/month) Many Lives Single Life 0 Essential Funds Discretionary Funds Comfort Size (# of personnel) Culture (% thriving on chaos vs. order) October 16, 2012 Copyright USC-CSSE 26

27 Architected Agile Approach Uses Scrum of Scrums approach Up to 10 Scrum teams of 10 people each Has worked for distributed international teams Going to three levels generally infeasible General approach shown below Often tailored to special circumstances October 16, 2012 Copyright 27 USC-CSSE

28 COCOMO II Productivity Ranges Scale Factor Ranges: 10, 100, 1000 KSLOC Development Flexibility (FLEX) Team Cohesion (TEAM) Develop for Reuse (RUSE) Precedentedness (PREC) Architecture and Risk Resolution (RESL) Platform Experience (PEXP) Data Base Size (DATA) Required Development Schedule (SCED) Language and Tools Experience (LTEX) Process Maturity (PMAT) Storage Constraint (STOR) Use of Software Tools (TOOL) Platform Volatility (PVOL) Applications Experience (AEXP) Multi-Site Development (SITE) Documentation Match to Life Cycle Needs (DOCU) Required Software Reliability (RELY) Personnel Continuity (PCON) Time Constraint (TIME) Programmer Capability (PCAP) Analyst Capability (ACAP) Product Complexity (CPLX) Productivity Range October 16, 2012 Copyright 28 USC-CSSE

29 Return On Investment (ROI) University of Southern California Value-Based Testing: Empirical Data and ROI LiGuo Huang, ISESE 2005 (a) % of Value for Correct Customer Billing Bullock data Pareto distribution Automated test generation (ATG) tool - all tests have equal value Customer Type (b) % Tests Run Value-Neutral ATG Testing Value-Based Pareto Testing October 16, 2012 Copyright 29 USC-CSSE

30 Value-Neutral Defect Fixing Is Even Worse 100 Pareto Business Value Automated test generation tool - all tests have equal value 80 % of Value for Correct Customer Billing Value-neutral defect fixing: Quickly reduce # of defects Customer Type October 16, 2012 Copyright USC-CSSE 30

31 Reuse at HP s Queensferry Telecommunication Division Time to Market (months) Year Non-reuse Project Reuse project 31 October 16, 2012 Copyright USC-CSSE

32 Product Line Engineering and Management October 16, 2012 Copyright USC-CSSE 32

33 Overfocus on Acquisition Cost C4ISR Contracts: Nominal-case requirements; 90 days to PDR 33 October 16, 2012 Copyright USC-CSSE

34 C4ISR Project C: Architecting for Change USAF/ESC-TRW CCPDS-R Project* When investments made in architecture, average time for change order becomes relatively stable over time * Walker Royce, Software Project Management: A Unified Framework. Addison-Wesley, October 16, 2012 Copyright USC-CSSE

35 250.00% % % Relative* Total Ownership Cost (TOC) ~5% architecture investment ~5% architecture investment % ~25% architecture investment 50.00% 0.00% Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5 Project A Project B Project C * Cumulative architecting and rework effort relative to initial development effort 35 October 16, 2012 Copyright USC-CSSE

36 Ilities in Tradespace Exploration: MIT Enabling Construct: Tradespace Networks Changeability More changeable (ie including flexible, adaptable, scalable and modifiable) Colored by outdegree For this plot, Ĉ=C Survivability Enabling Construct: Epochs and Eras Value Robustness Set of Metrics 2012 Massachusetts Institute of Technology seari.mit.edu 36

37 Architecture-Based Attribute Trades: Flexibility Flexibility Arch. Example Synergies (RT-18a) Conflicts Strategy High module cohesion; Low module coupling Service-oriented architecture Autonomous adaptive systems Modularization around sources of change Interoperability Reliability Composability, Usability, Testability Affordability via task automation; Response time Interoperability, Usability, Reliability, Availability High Performance via Tight coupling High Performance via Tight coupling Excess autonomy reduces human Controllability Extra time on critical path of Rapid Fielding Multi-layered architecture Reliability, Availability Lower Performance due to layer traversal overhead Many built-in options, entry points Functionality, Accessibility Reduced Usability via options proliferation; harder to Secure User programmability Usability, Mission Effectiveness Full programmability causes Reliability, Safety, Security risks Spare/expandable capacity Performance, Reliability Added cost Product line architecture, reusable components Cost, Schedule, Reliability Some loss of performance vs. optimized stovepipes October 16, 2012 Copyright USC-CSSE 37

38 Value/Risk-Based Tradespace Analysis - Early Startup: Risk due to low dependability - Commercial: Risk due to low dependability - High Finance: Risk due to low dependability - Risk due to market share erosion 1 Combined Risk Exposure 0.8 Market Share Erosion 0.6 RE = P(L) * S(L) 0.4 Sweet Spot Early Startup Commercial High Finance VL L N H VH RELY COCOMO II: Added % test time COQUALMO: P(L) Early Startup: S(L) Commercial: S(L) High Finance: S(L) Market Risk: RE 07/08/09 m 38 USC-CSSE

39 Magnitude of Overrun Problem: DoD October 16, 2012 Copyright USC-CSSE 39

40 Magnitude of Overrun Problem: Standish Surveys of Commercial Projects Year Within budget and schedule Prematurely cancelled Budget or schedule overrun October 16, 2012 Copyright USC-CSSE 40

41 Some Frequent Overrun Causes Conspiracy of Optimism Effects of First Budget Shortfall System Engineering Decoupling of Technical and Cost Analysis Overfocus on Performance, Security, Functionality Overfocus on Acquisition Cost Assumption of Stability Total vs. Incremental Commitment October 16, 2012 Copyright USC-CSSE 41

42 The Conspiracy of Optimism and The Cone of Uncertainty October 16, 2012 Copyright USC-CSSE 42

43 Effects of First Budget Shortfall: Added Cost of Weak System Engineering Calibration of COCOMO II Architecture and Risk Resolution (RESL) factor to 161 project data points October 16, 2012 Copyright USC-CSSE 43

44 Assumption of Stability vs. Rapid Change Need evolutionary/incremental vs. one-shot development 4x 2x Uncertainties in competition, technology, organizations, mission priorities 1.5x 1.25x Relative Cost Range x 0.8x 0.67x 0.5x 0.25x Concept of Operation Rqts. Spec. Product Design Spec. Detail Design Spec. Accepted Software Feasibility Plans and Rqts. Product Design Detail Design Devel. and Test October 16, 2012 Phases and Milestones Copyright USC-CSSE 44