Webinar: Sizing, Cost, Schedule and Risk for Software: A 10 Step Process. September 22, :00 PM 1:30 PM

Transcription

1 Webinar: Sizing, Cost, Schedule and Risk for Software: A 10 Step Process September 22, :00 PM 1:30 PM ITMPI005 1

2 Daniel D Galorath Founder & CEO Galorath, Inc. dgalorath@galorath.com Michael Milutis Director of Marketing Computer Aid, Inc. (CAI) Michael_milutis@compaid.com 2

3 About Galorath Galorath began as a consulting firm in 1979, committed to assisting government and industry to improve their software and hardware development products and program management. SEER by Galorath was initially envisioned by a veteran project engineer and a team of senior scientists and analysts who experienced first hand the shortcomings of manual estimation processes: Information gaps, inconsistent processes, changing requirements, competing demands, and outside pressure borne of wishful thinking. Galorath has invested over two decades of research and development helping organizations better plan and control project costs, quality, duration and risk. Leveraging sophisticated modeling technology and project-applicable knowledge bases, SEER solutions are proven to accurately replicate realworld project outcomes more quickly and with much greater accuracy than any traditional estimating methodologies. 3

4 CAI Achieves IT Operational Excellence 4

5 PDU CREDITS FOR THIS WEBINAR The Project Management Institute has accredited this webinar with PDUs 5

6 NOW AVAILABLE! ONLINE WEBINAR RECORDINGS ANYTIME ACCESS! WWW. ITMPI.ORG / LIBRARY 7 Day Free Access For All Recordings ITMPI 6

7 Key Points Software is critical to systems and is getting more complex Software estimating processes are core to reducing software risk Software estimation & metrics can provide program managers significant information 2010 Copyright Galorath Incorporated 7

8 An Estimate Defined An estimate is the most knowledgeable statement you can make at a particular point in time regarding: Effort / Cost Schedule Staffing Risk Reliability Estimates more precise with progress A WELL FORMED ESTIMATE IS A DISTRIBUTION 2010 Copyright Galorath Incorporated 8

9 People, Process, Technology Are Keys [Source CMMI Tutorial] Everyone realizes the importance of having a motivated, quality work force but......even our finest people can t perform at their best when the process is not understood or operating at its best. PEOPLE PROCESS TECHNOLOGY Major determinants of cost, schedule, & quality 9

10 Estimation Process Is Important to Successful Projects Estimate & Plan Quantitative Project Management Measure & Analyze Monitor & Control A Foundation of Risk Management 10

11 Poor Estimates Effects on Projects Inaccurate estimates significant impact on project success: Poor implementations Critical processes don t scale Emergency staffing Cost overruns caused by underestimating project needs Lack of well defined objectives, requirements, & specifications, results in creeping scope resulting in: Forever changing project goals Frustration Customer dissatisfaction Cost overruns and missed schedules Project Failures Incorrect estimates / bad plans are a root cause of subsequent program risk Estimating & Planning are key to software project success 11

12 Bad Estimate Root Causes: Software Managers View From (ISPA) Parametric Estimating Handbook: Optimistic (low) estimates of size Failure to control scope creep Unusable heritage code replaced with new code Development environment (negative) impacts on productivity Personnel (negative) impacts on productivity Lack of requirements definition = volatile baseline Senior Software PM Experience Why An Estimate Went Awry: Seldom keep the same staff through an entire project Tools break or I m told.. this tool will crank up our productivity. Customer is always WRONG They change their mind often! The big bosses try to fix it with their vast experience Failure to understand Brook s Law. Adding staff to a late software project makes it later. Why should we care: Their bad estimates can yield our failed program 12

13 10 Step Software Estimation Process: Consistent Processes = Reliable Estimates 1. Establish Estimate Scope 10. Track Project Throughout Development 2. Establish Technical Baseline, Ground Rules, Assumptions 9. Document Estimate and Lessons Learned 8. Generate a Project Plan 3. Collect Data 7. Quantify Risks and Risk Analysis 4. Estimate and Validate Software Size 6. Review, Verify and Validate Estimate Prepare Baseline Estimates

14 Level 0 Estimation Organizational Maturity V1.7 Informal or no estimating Manual effort estimating without a process Level 1 Direct Task Estimation Spreadsheets Ad Hoc Process Level 2 Formal Sizing (e.g. function points) Direct Task Estimation Simple model (Size * Productivity) or informal SEER Use Some measurement & analysis Informal Process Level 3 Formal Sizing Robust Parametric estimation (SEER) Estimate vs. actual capture Formalized Multiple Estimate Process Rigorous measurement & analysis Parametric planning & Control Risk Management Repeatable process Level 4 Formal sizing Repeatable process Robust parametric estimating (SEER) Rigorous measurement & analysis Parametric estimation with tracking & control Risk Management Process improvement via lessons learned Level 5 Formal sizing Repeatable process Robust parametric estimating (SEER) Rigorous measurement & analysis Parametric estimation with tracking & control Risk Management Continuous process improvement Why should we care? Maturity is related to estimate viability Better estimation process more likely to be successful in execution 14

15 Step One: Establish Estimate Scope and Purpose Define and document estimate expectations, scope & purpose Provides a baseline against which to gauge the effect of future changes Reduces misunderstandings & contradictory assumptions Estimate should be considered a living document As projects change, data changes or new information becomes available, it should be documented and factored into the estimate in order to maintain the project s integrity 15

16 Define What s Included in the Estimate Effort breaks down into What functionality will be developed or maintained How will it be done? Activities & phases Who will do the work? Personnel labor categories 16

17 The Importance of Scope Nature of estimate ROM or precision? Most likely, worst case, or should cost scenario? Cross-check or target costing exercise? Acquisition or life-cycle? Risk analysis included? Nature of system Number of subsystems: flight, ground... Number of WBS elements Level of indenture (CSCI, CSC, CSU) Number of releases / incremental builds Nature of code All new, any reuse? Effort applied to reused code: modify, I&T, etc. Any COTS or GOTS to be integrated? Any code generators to be used? Know what you need to do before you need to do it 17

18 Step Two: Establish Technical Baseline, Ground rules, & Assumptions Functionality included in the estimate or range must be established (Technical Baseline) If detailed functionality is not known, groundrules and assumptions state what is and isn t included in the estimate Issues of COTS, reuse, and other assumptions should be documented as well Groundrules and assumptions form the foundation of the estimate Although at early stages they are preliminary and therefore rife with uncertainty, they must be credible and documented Review and redefine these assumptions regularly as the estimate moves forward 18

19 Technical Baseline Describe each work element Core functionality Key qualitative modeling inputs Size, if available Identify multiple builds/releases (if any) Reference or excerpt available documentation Provide rationale & justification for BOE (Basis Of Estimate) For iterative development identify overall range of project technology 19

20 Ground rules & Assumptions Ground rule: given requirement of the estimate (e.g. software must support Windows and Linux) Be sure to provide source Assumption: assumed to scope estimate Be sure to provide source and substantiation What s known, what s unknown Anything relating to scope What s included, what s excluded Anything relating to modeling inputs Who you interviewed and when What you learned 20

21 Step Three: Collect Data Data Collection Process Key Considerations 1. Motivate potential data providers to participate 2. Avoid nondisclosure agreements containing clauses requiring exclusivity or destruction of data if you can 3. Provide data collection forms and instructions beforehand, in both hard copy and electronic formats 4. Provide clear definitions but recognize providers may not read them 5. Identify which data are required, highly desirable or desirable 6. During the face-to-face interview confirm data is realistic and valid 7. Grade to indicate confidence 8. Normalize data via well-documented process & keep both the raw and normalized data 21

22 The Error of Causal Analysis Creating a False Association 22

23 The Error of Causal Analysis Is often encountered in data analysis! This point does not include SW Testing Effort System and SW Requirements were done separately Code and Unit Test Only There is no correlation that estimate of size X is an equivalent of historical projects of about X. 23

24 Step Four: Software Sizing Spend time on sizing Include reuse analysis and delivered sizing Estimate least, likely, most range Some widely used methods of estimating product size are: Expert opinion Analogy Formalized methodology Statistical sizing Provides a range of potential sizes that is characterized by least, likely, and most 24

25 Size Has Many Dimensions Newly developed Consulting Redesign Existing Glue Code Retest Off the shelf Configuration Reimplement 25

26 Step Five: Prepare Baseline Estimate 1. Trained, experienced, and skilled people should be assigned to size the software and prepare the estimates 2. It is critically important that they be given the proper technology and tools 3. Project manager must define and implement a mature, documented, and repeatable estimation process 26

27 Estimation Methods 1 of 2 Model Category Description Advantages Limitations Guessing Off the cuff estimates Quick Can obtain any answer desired No Basis or substantiation No Process Usually Wrong Analogy Compare project with past similar projects. Estimates are based on actual experience. Truly similar projects must exist. Expert Judgment Consult with one or more experts. Little or no historical data is needed; good for new or unique projects. Experts tend to be biased; knowledge level is sometimes questionable; may not be consistent. Top Down Estimation A hierarchical decomposition of the system into progressively smaller components is used to estimate the size of a software component. Provides an estimate linked to requirements and allows common libraries to size lower level components. Need valid requirements. Difficult to track architecture; engineering bias may lead to underestimation. 27

28 Estimation Methods 2 of 2 Model Category Description Advantages Limitations Design To Cost Uses expert judgment to determine how much functionality can be provided for given budget. Easy to get under stakeholder number Little or no engineering basis. Simple CER s Equation with one or more unknowns that provides cost / schedule estimate Some basis in data Simple relationships may not tell the whole story Historical data may not tell the whole story Comprehensive Parametric Models Perform overall estimate using design parameters and mathematical algorithms. Models are usually fast and easy to use, and useful early in a program; they are also objective and repeatable. Models can be inaccurate if not properly calibrated and validated; historical data may not be relevant to new programs; optimism in parameters may lead to underestimation. 28

29 Manual Estimates Human Reasons For Error Desire for credibility motivates overestimate behavior (80% probability?) Must spend all the time to be reliable Better approach force 50% probability & have buffer for overruns Technical pride causes underestimates Buy-in causes underestimates 50% 80% 29

30 Knowing Software Planning Possibilities Is Critical To Success For a given Size, Technology, Complexity & Probability Effort (person-months) Minimum Time Impossible Inefficient Reasonable Plan Range Optimal Effort SEER Software Equations Elapsed Calendar Time (months) 30

31 Software Estimation Basic Model & Associated Metrics Effective Technology Cte Effort K Effective complexity D Size St ReuseDIT Size Se (work units) People Staff m & Constraints Stakeholder Requirements Start Technology Process Calendar Time Defects Count (Qi Qr) Software Development Process Development Legacy, Maintenance Specifics & Constraints and/or Block Changes As Redevelopment Delivered Software Finish Maintenance/ Block Change Development Process Size (Effective Se & Total St) On-going Iterations of Effort (ACWP or Spent) Progress (BCWP or Earned Value) Defects (Qi Qr ) Growth (Sg) 31

32 Staff Level (FTE people) Cost & Time Penalty SEER-SEM: Avoid Death Marches & Failed Projects By Applying Brooks Law Unaccomplished Work Planned Schedule or Cost Optimal Staffing Schedule Slip Delivery Elapsed Calendar Time (months) Level Staffing Actual Delivery Effective Staffing Staffing Beyond Plan Overstaffed Understaffed 32

33 Understand Project Total Ownership Costs Up Front Most Projects Spend Low During Maintenance Staff Vs M aintenance Rigor staff hours per month develop Rigor vhi+ Rigor nom Rigor vlo Time 33

34 Generate the Estimate Using chosen methodology and tool, do a first run Never report preliminary results! Focus on the inputs Verify completeness Verify accuracy Focus on the outputs Sanity check for reasonableness, completeness What s driving the estimate? Top ten parameters Use fresh eyes to review Ask a colleague for help Set aside overnight 34

35 A cross check estimate prepared using a different methodology and/or tool may be required Understand the usage of the cross check methodology and/or tool Keep the focus on the estimate, not on the tool Perform Cross Checks 35

36 Scope project Depth & breadth Use project plan Data collection Spend most time on sizing Document sources Tech baseline Document all critical inputs GR&A, estimating process Prepare estimate Verify inputs Sanity check outputs Use fresh eyes Never deliver preliminary results Process Checklist 2010 Copyright Galorath Incorporated 36

37 Step Six: Quantify Risks and Risk Analysis Risk can produce loss of time, quality, money, control, understanding, etc. Approximate the probability that the event will occur Determine how risk can be mitigated Risk control involves actions taken to reduce or eliminate a risk Risk management identifies & addresses internal & external threats Sizing & estimating potentially have dramatic negative effects If risk foreseen & causes acted upon effects can be mitigated Although cost, schedule, & performance risks are interrelated, they can also be analyzed independently Risks must be identified specifically to be manageable Statistical risk/uncertainty analysis should be a part of schedule & effort estimation process 37

38 Understanding Risk and Uncertainty is Essential To Project Management 2010 Copyright Galorath Incorporated 38

39 Software & IT Systems Are About Business Value Value Return on investment Internal rate of return Net present value Intangibles Cost Development Maintenance IT infrastructure IT services Other direct costs

40 The Components of Measurement for Business Decision Making Measurement of Actuals Quantification of Risk Estimation of Cost Estimation of Value 2011 Copyright Galorath Incorporated

41 Step Seven: Estimate Validation and Review Ideally, validation performed by one who was not involved in generating the estimate Assess estimate assumptions Ensure ground rules are consistently applied Rigorous validation process exposes faulty assumptions, unreliable data and estimator bias Provides clearer understanding of risks With isolated problems at their source, containment can occur & a more realistic picture of project success requirements Failing to validate the estimate may result in much greater downstream costs, or even a failed project 2010 Copyright Galorath Incorporated 41

42 Compare Parametrics With Metrics and Sanity Checks Works with common repository Shows actual data, ranges, and correlations Plots parametric estimates and contrasts with data points Plots actual data and / or trends 42

43 Step Eight: Generate A Project Plan Includes allocating estimate cost & schedule to a function & task-oriented work breakdown structure Issues Inexperience evaluating decisions long term impacts Lack necessary information Believe time to develop the information will make the project suffer Decisions based on what management wants to hear Good manager understands project realities even if it is not what higher management wants Must explain reality in language his managers can understand Problem managers either lead a project to an unintended conclusion or, worse, drift down the road to disaster Software management & planning problems have been recognized as leading causes of project failures Bad management decisions Incorrect focus Destructive politics 43

44 Step Nine: Document Estimate and Lessons Learned Document upon estimate complete AND project complete Document the pertinent information Record lessons learned Provides evidence that your process was valid The estimate was in good faith Can substantiate or calibrate your estimation models Document missing or incomplete information & risks, issues, and problems the process addressed & any complications that arose Document key decisions made during the estimate & results Document dynamics that occurred during the process e.g. Interactions of your estimation team Interfaces with clients Trade-offs made to address issues identified during the process Conduct lessons-learned session ASAP after completion while participants memories are still fresh Every software project should be used as an opportunity to improve the estimating process 44

45 Step Ten: Track Project Throughout Development Refine estimates throughout project Use estimates as basis for performance measurement & project control Monitor actual effort & duration of tasks and/or phases against planned values Apply earned value techniques along with parametric estimation to help ensure success Evaluate defects & growth in addition to simple earned value 45

46 Use Earned Value To Quantify Progress Versus Effort The main concern of EVM is what has been accomplished in a given time and budget, versus what was planned for the same time and budget A project is generally deemed healthy if what has been accomplished is what was planned, or more A project is deemed unhealthy if accomplishment lags expectations Definition: Earned Value = Budgeted Value for the work accomplished (what you got for what it cost you) $ Healthy Budget $ Unhealthy Budget EV EV Time = Now Time = Now 46

47 Defects and Growth Impact Software Process Heath and Status Indicator shows status and trends from the previous snapshot Thresholds are user definable Track defect discovery and removal rates against expected rates Increased defect reporting rate shows a worsening trend 47

48 Parametric Project Monitoring & Control Provides Performance Measurement aspects of ANSI/EIA-STD-748 Adds Performance Measurement (Earned Value) methods to parametric estimation model Accepts progress & expenditure inputs Provides cost, schedule, and time variances Provides cost, schedule, & time indices Performance-based cost & schedule Estimate at Completion Displays health and status indicators 48

49 Estimation Lessons Learned Estimating process is essential Estimate should be the basis of the plan Measure early Before trouble Early under-budget milestones not necessarily a good sign Programs that skimp on the early, upfront planning - requirements and design work - will most likely be in trouble later Estimation, planning, tracking, controlling then using the information to do better next time EVM shouldn t be used alone Other metrics are necessary to be kept in concert with the EVM metrics in order to keep a project on track Recognize that you have a problem 49

50 7 Characteristics of a Dysfunctional Software Project (Source: Mike Evans, et al.) Based on 350 projects: Failure to Apply Essential Project Management Practices Unwarranted Optimism and Unrealistic Management Expectations Failure to Implement Effective Software Processes Premature Victory Declarations Lack of Program Management Leadership Untimely Decision-Making Lack of Proactive Risk Management 50

51 Conclusions Software projects are manageable Viable estimate forms a basis for management The 10 step process provides a repeatable approach to estimation process You can help ensure useful results by adopting a process that is standardized and repeatable Measurement is a key part of the estimation process Beware of using simple measurements to drive estimates 51

52 Questions? 52

53 CAI Sponsors The IT Metrics & Productivity Institute: Clearinghouse Repository of Best Practices: Weekly Educational Newsletter: / SUBSCRIBE Weekly Webinars Hosted by Industry Leaders: / WEBINARS ACCESS WEBINAR RECORDINGS ANYTIME AT / LIBRARY Follow Us on TWITTER at / ITMPI Join Our Network on LINKED IN at LINKEDIN Follow Us on FACEBOOK at FACEBOOK Find Out About Our CONFERENCES at EVENTS 53

54 Daniel D Galorath Founder & CEO Galorath, Inc. dgalorath@galorath.com Michael Milutis Director of Marketing Computer Aid, Inc. (CAI) Michael_milutis@compaid.com 54