Affordability Analysis: The Role Process, Cost and ROI Modeling In Improved Program Performance SEI October 2012

Transcription

1 Affordability Analysis: The Role Process, Cost and ROI Modeling In Improved Program Performance SEI October 2012 Galorath Incorporated Daniel D. Galorath: Founder & CEO Copyright Galorath Incorporated 2012

2 Key Points Viable affordability decisions yield project achievements Positive ROI when decisions use affordability modeling Parametric Models provide affordability decision support Copyright Galorath Incorporated 2

3 Cost Overruns Are Everywhere GAO: Staggering cost overruns dwarf modest improvements in Defense acquisition R&D costs of weapons programs increased 42% over original estimates Average delay 22 months in delivering initial capabilities Evolving technical requirements Shortage of qualified government staff to manage "Every dollar of cost growth on a DoD weapon system represents a lost opportunity to pay for another national priority" The Collins Class submarine program: Murphy was an optimist BG Group shares hit by $5.4B cost overruns on Australian Liquified Natural Gas project

4 Software Is A Key Risk Item In Weapons Systems Navy Mobile User Objective Satellite Communication System delays to the Joint Tactical Radio System, a set of software-defined radios causes advanced MUOS capabilities to be drastically underused GAO GAO identified 42 programs at risk for cost & schedule 1. military requirements changes 2. software development challenges 3. workforce issues National Institute of Standards and Technology (NIST) Software defects cost nearly $60 Billion Annually 80% of development costs involve identifying and correcting defects Software, not Hardware or technology readiness levels were called out

5 Example: Project Cost Alone Is not The Cost of IT Failure (Source: HBR) Case Study: Levi Strauss $5M ERP deployment contracted Risks seemed small Difficulty interfacing with customer s systems Had to shut down production Unable to fill orders for 3 weeks $192.5M charge against earnings on a $5M IT project failure IT projects touch so many aspects of organization they pose a new singular risk

6 An ROI Analysis of A New System: Should We Fund This? Can we do better? Will stakeholders tolerate a loss for 3 years? What is the risk?

7 Bad Estimates Are A Root Cause of Project Failure An estimate is the most knowledgeable statement you can make at a particular point in time regarding: Effort / Cost Schedule Staffing Risk Reliability Estimates more precise with progress A WELL FORMED ESTIMATE IS A DISTRIBUTION 2012 Copyright Galorath Incorporated 7

8 Poor Project Performance Often Points Back To Inadequate Estimating When performance doesn t meet the estimate, there are two possible causes: 1. Poor performance 2. Poor estimates In the software world, we have ample evidence that our estimates stink, but virtually no evidence that people in general don t work hard or intelligently enough. - Tom DeMarco, Why Does Software Cost So Much? Why should we care: Good people working with poor estimates cause project challenges Copyright Galorath Incorporated 8

9 Greatest Leverage Is In Tackling Inherent and Structural Costs 2010 Copyright Galorath Incorporated 9

10 2010 Copyright Galorath Incorporated 10

11 US Better Buying Power Initiatives June 28, 2010 Mandate September 14, 2010 Guidance November 3, 2010 Implementation Five Specific Areas of Concern: Target Affordability and Control Cost Growth Reduce Non-Productive Processes and Bureaucracy Incentivize Productivity and Innovation in Industry Promote Real Competition Improve Tradecraft in Services Acquisition

12 Affordability Initiatives With Should Cost and Will Cost Will Cost Cost Initiatives (Applied practices & improvements) Performance - = Should Cost Performance Many View Bottoms up estimates as the requirement for Should Cost / Will Cost Analysis But parametrics can do analysis faster as well as provide more tradeoffs 2012 Copyright Galorath Incorporated 12

13 System Description (Parametrics Can Estimate More, Earlier) Adapted from CEBOK If you can t tell me what it is, I can t tell you what it costs. -Mike Jeffers If you can tell me the range of what it might be, I can tell you the range of cost, schedule & probability. -Dan Galorath

14 Affordability Process Step 1. Procure Key Performance Parameters that are inviolate Step 2. Identify Affordability Goals & Figures of Merit (development, life cycle, payback, ROI, NPV, kill ratio, Budget constraints, etc.) Step 3. Gather Requirements, Features, Performance Step 4. Define Baseline Alternatives Step 5. Perform Technical Design Analysis for Each Alternative Step 6. Perform Cost Schedule Analysis of Each Alternative Step 7. Assess Benefits Based on Figures of Merit Step 8. Perform Probabilistic Risk Analysis Step 9. Assess Alternatives & Select Optimal Alternative Step 10. Document Analysis and Lessons Learned

15 Should Cost: Trade Study Flow Requirements And Features Analysis Preliminary Design Process Human Factors Security Reliability Availability Survivability Other Factors Alternative 1 Alternative 2 Alternative 3 Alternative 4 Supportability Testability Producibility Reuse Transportability SEER Assessment Iterate Performance Schedule Risk Assessment Life Cycle Cost 1 No Yes Selection Process OK? Optimized? Cost Performance Schedule Risk Bottoms Up Estimation as Required 2012 Copyright Galorath Incorporated 15

16 Top-Down versus Bottom-Up Estimating (Adapted From: Project Management: the Managerial Process, McGraw Hill 2011) Condition Macro Level / Parametric Strategic decision X making Internal, Small Project x Fixed price contract Customer wants detail Unstable scope x Bottoms up can be more accurate IF All work elements are captured and Micro level / Bottoms up Inter WBS work is not forgotten Parametrics can be more accurate since it won t forget items Use parametrics AT LEAST for analysis of alternatives x x 2010 Copyright Galorath Incorporated 16

17 Complex Parametric Modeling: Repeatable Based on Program Parametric Estimating Mathematical relationships between anticipated characteristics and effort, schedule, risk, etc. & Technology, programmatics, cost, schedule, To estimate Development Production Effort / cost / Total Cost of Ownership Schedule Manufacture Modification Deployment of end items Much more Part of a repeatable process Traditional Estimating Many techniques from: Guessing Simple Analogy Spreadsheets Bottoms up Quantity x price Etc. Generally manual in nature Most are not repeatable

18 Firm Fixed Price? Feel lucky? What is likely to happen Understand the risk before you commit! 18

19 Dealing With the Problem of Assumptions Assumptions are essential but Incorrect assumptions can drive an estimate to uselessness Use an assumption verification process 1. Identify assumptions 2. Rank order assumptions based on estimate impact 3. Identify high ranking assumptions that are risky 4. Clarify high ranking, high risk assumptions & quantify what happens if those assumptions change 5. Adjust range of SEER inputs to describe the uncertainty in assumptions 2012 Copyright Galorath Incorporated 19

20 Key Points Viable affordability decisions yield project achievements Positive ROI when decisions use affordability modeling Parametric Models provide affordability decision support Copyright Galorath Incorporated 20