BAE Systems Software Estimation Steve Webb BAE Systems Estimating Focus Group Chairman Engineering Estimation & Measurement Manager 22 April 2008 1
Background BAE Systems Estimating Focus Group covers the whole of BAE except North America produced BAE Systems Engineering & Software Estimation Handbooks, Guides, Estimation Workbooks, BAE Systems Code Counter tool, UK BAE Systems COSYSMO model BAE Systems Integrated System Technologies () covers defence, homeland security & complex, mission-critical solutions across 15 UK sites with around 4,000 staff software projects: range from 1 person to many hundreds of effort-years of development per project large systems of around 1,000,000 hand-written logical Source Lines Of Code plus auto-generated code has a CMMI initiative & some parts are CMM level 3 2
Software Bids bid estimates do not contain any hidden contingency Accurate estimates are essential to reduce the amount of contingency to win orders Bid estimates are produced by engineering staff for the bid team Bids are independently reviewed in 5 stages: 1. Estimation Review for estimation completeness, accuracy, etc done by Engineering 2. Cost Review by the company Chief Engineer (or team) 3. Technical Bid Review (meets Win Strategy, Target Cost, ITT) 4. Phase 2A Review the chair is independent of the Bid team 5. Commercial Review 1 & 2 are performed by Engineering staff who understand software & estimation 3
Software Estimation Process has a Common Software Engineering Process (CSEP) Only 1 Software Measurement process Only 1 Software Estimation process & all software bids use it This presentation will concentrate on Software Estimation process Next 3 slides show 3 of the 12 html Software Estimation process pages All statements requiring action in this process are really shalls statements 4
5
6
7
Estimation Methods: Analogy (e.g. within 10%) - a preferred method Expert Judgement Top-down Bottom-up Parametric Models (e.g. within 20%) e.g. COCOMO II, own tools Proxy (e.g. Use Case points) Used to create size & effort estimates 8
Estimate at least 2 different ways Estimate at least 2 ways at the CSCI/component (i.e. Computer Software Configuration Item) level & combine them into 1 overall estimate E.g., estimate 1 = Top-down (analogy size based), estimate 2 = Bottom-up Bottom-up estimate is broken down into s/w lifecycle phases, work products, support & management activities use generic software WBS document Two staff independently estimating the same way is only deemed as 1 way Using 3 points estimation (min, most likely, max) is part of estimating 1 way At bid phase more accuracy is achieved by estimating 2 different ways, using single point estimates, than estimating just 1 way with 3-points One estimate must be derived from work product size e.g. number of: Logical Source Lines Of Code Use Cases points Document lines (e.g. Design, Test documents) 9
Bid Estimates Reconcile these estimates into 1 final estimate Estimate 1: Top-down Size Analogy (inc. 3-points) Estimate 2: Bottom-up (inc. 3-points) Estimate n Final Effort Estimate (inc. 3-points) 50%, 70%, 90% development uncertainty points 10
Estimate Reconciliation Resolved at the component level Find out why the difference in predicted effort e.g. check estimates cover same scope, check level of detail, check expected accuracy Ensure within bounds of each other e.g. within 10% of median If estimate(s) still outside these bounds then e.g.: estimate additional estimated way(s) improve the accuracy by estimating in more detail increase use of Delphi technique No hard rules on how to combine the estimates Sometimes use Wideband Delphi but normally use Delphi group meeting 11
Development Uncertainty Risk estimates use Triangular distribution Development uncertainty estimates typically use Beta distribution & deterministic formulae - this is as accurate as Monte-Carlo Mandated to estimate 50%, 70%, 90% development uncertainty effort pts If high uncertainty then Monte Carlo 50%, 70%, 90% schedule pts are done Management may decide more than the above is required Technical & Management Contingency budgets hold development reserve Technical Contingency budget takes into account the amount of development uncertainty and the amount of risk 12
David Henry study & Software Estimation: Perfect Practice Makes Perfect by David Henry Read the June 2002 CrossTalk article by David Henry Engineers used 3-point estimates & deterministic Beta distribution spreadsheet which contained history and comments - very useful feedback loop Accuracy of estimates dramatically improved doing weekly estimation Average % difference between estimates and actual task completion times Month 1 - typically were under-estimates Variance in group Month 1 75% 25% - 150% Month 3 35% Month 6 25% Some staff (inc. subject matter experts) will significantly under-estimate the max pt without practice can result in estimated 90% pt becoming actual 70% pt See next slide for the software engineer spreadsheet for estimating development uncertainty using a deterministic formulae for effort/cost 13
3-point Spreadsheet Three Point Estimation Spreadsheet (BMS Id 19608) Overall Information: Overall 50% pt = sum of 50% pts = 4,991.7 Feed-back information: Total Expected - completed activities only = 1,650.0 Total Actual - completed activities only = 1,962.0 Total Difference (compared to Expected) = 19% Task Name Minimum Most Likely Maximum Individual 50% Point Actual % Difference Task 1 500 900 2000 1016.7 1221.0 20% Task 2 Task 3 300 Uses the Traditional Beta Distribution 200 350 250 450 450 358.3 275.0 376.0 365.0 5% 33% Task 4 Task 5 Use the spreadsheet with either 200 effort or 250size (not 350 both) 258.3 2500 3000 4000 3083.3 0.0 0.0 0% 0% engineers use this spreadsheet to monitor their estimation capabilities 14
Estimating Size Size (or the size of the changes) is a big driver of costs Staff tend to underestimate size resulting in underestimated effort figure Key size items estimated by 2 staff independently of each other i.e. Delphi Analogy data obtained from Engineering Size & Productivity database All projects must write a detailed measurement report comparing the bid estimate with actuals It must include size, effort, productivity, at project & component level & include performance factors, software lifecycle effort percentages BAE Systems Software Size Estimation workbook estimates 10%, 50%, 70% and 90% size and effort development uncertainty points 15
BAE Systems Code Counter Tool One of the most advanced tools of its type Used across the whole of BAE including CMMI level 5 companies Tool was developed by DoD tasked USC to produce a SLOC difference tool Counts 16 languages and different types of Assembler Counts logical SLOC and physical SLOC Counts amount of new files, deleted files Counts amount of added, changed, unmodified, deleted SLOC within a file Counts at file, component or any directory level Can exclude counting source test files by using ignore file/folder facility Has a self-check counting mechanism Has GUI and Batch mode Easily installed by the engineer 16
Using this Tool counts new, added, adopted, adapted & deleted SLOC per component Using actual effort it has been easy to populate the Engineering Size & Productivity database New Malden very large projects e.g. Type 23, Type 45, Astute keep measurements at the component level In general the data confirms the COCOMO II Reuse Model as a valid model Model breaks down on very large projects with many small modifications i.e. COCOMO II AA should be zero but we set it to non-zero 17
Logical SLOC production monitored by DataDrill Project Manual Logical SLOC Size 18
BAE Systems Software Size Estimation workbook Estimates effort for new, adapted and adopted work Uses COCOMO II Reuse Model to estimate adopted & adapted productivity It requires input of size (e.g. SLOC, Use Case Pts) & new work productivity 19
BAE Systems Software Size Estimation workbook example Whole workbook is far too big to get on to a slide 10,167 is an exact figure as counted by BAE Systems Code Counter E.g. component with new work (i.e. blue) plus adapting 10,167 logical SLOC with e.g. Reqs Rework = 2%, Code Rework 10%, Integration Test = 15% Functional Name (cells A11 to A45) Size min Size most likely Size max Size 50% Reuse Min Reuse Most Likely Reuse Max Reuse 50% Est. % Reqs Anal. Rework Est. % Arch. Design Rework Est. % Detailed Design Rework Est. % Code & Unit Test Rework Est. % Int. & Test Rework Est. % Fn Proving (FAT) Rework AAF TOTAL 2,000 2,500 3,500 2,583 10,167 10,167 10,167 10,167 Component example 2,000 2,500 3,500 2,583 10,167 10,167 10,167 10,167 2% 2% 10% 10% 15% 20% 10.2 % 20
Use appropriate historical data It is important to use appropriate data for Analogy estimates Productivity will vary tremendously on different projects for valid reasons Logical SLOC size alone accounts for 2 times difference in productivity Type of modification e.g. new code compared to changed code can show 2 times difference Performance factors (inc. product complexity) accounts for 3 times difference So a similar component is used in the analogy comparison 21
Performance Factors BAE Systems Software Size Estimation workbook requires to know what changes to performance factors have occurred since historical data Most factors are described in the COCOMO II Model Definition Manual Some of the key factors are: produced similar systems before applications, tools, programming, platform experience staff capabilities product (technical) complexity amount of reuse component coupling schedule compression the requirements quality (completeness, stability) the effects of multi-site working Often staff estimate the differences in performance factors without using COCOMO II 22
Performance Factors Example Performance Factor Applications Experience Programming Experience Schedule Duration Base Estimated Modifier Comments Effect % decrease 10% 0.9 Since this component was first developed staff applications experience has improved. decrease 10% 0.9 Now more experience of C# increase 15% 1.15 This component is on the critical path. It is required to be produced in 8.5 months not the typical 10 months. TOTAL 0.93 0.93 = 0.90 * 0.90 * 1.15 If 100 units of effort were spent on original project it would have taken 90 units with the applications experience staff they now have The above is multiplicative and tries to behave like parametric tools TOTAL of <1 will mean an increase of productivity If original productivity was 2.0 SLOC/hr then new prod. = 2.0 / 0.93 = 2.15 SLOC/hr If a large no. of factor differences between old and new component then a COCOMO II estimate is produced instead 23
24
Review & Record Estimate Independently reviewed by someone who understands s/w & estimating Checks it is complete & compliant with estimates objectives & constraints Estimate is accurate given purpose of estimate (e.g. ROM, fixed price) Confirm it is traceable from top to bottom level It has no overlaps or inconsistencies internally or with other estimates All estimating figures (size, effort, cost, schedule duration, probability pts, expected accuracy) look reasonable Check rationale, assumptions, justifications Check all funding issues & dependencies are explicit Confirm risks been identified & risk estimates are adequate & appropriate Confirm risk mitigation actions budget is in the development estimate Identify & record any issues to be resolved, additional information to be supplied and/or questions to be answered prior to approval 25
Software Functional Manager (or delegated person): Independently reviews the bid estimate Records bid information in the Software Estimates Register Ensures bid Estimation Checklist is completed by estimator & reviewer Checklist helps estimators & reviewer and acts a sign-off Confirms all outstanding review actions been cleared Ensures the bid estimate is under configuration control 26
detailed measurement definitions e.g. How to count Size (e.g. Logical SLOC size, Use Case points) How to convert auto-generated SLOC into logical SLOC size How to count effort - what is software work, what is systems work, what is included in the software lifecycle Clear what is risk and what is uncertainty Without detailed definitions staff will estimate differently 27
Estimation Measurement report data contains: Bid estimated size and actual logical SLOC size At project level & component level record the actual amount of new, added, changed & unmodified logical SLOC New = new component Predicted & actual logical SLOC/hr productivity project & component level Record the reasons for the productivity (+ve & -ve) Software Lifecycle Breakdown, e.g. % spent on requirements, design, coding, integration & system test, management, configuration management Defect density per component Defect leakage table Amount of Rework in-phase and out-of-phase Amount of unpaid requirements growth & volatility 28
Summary Follow a well defined estimation process Use Analogy as much as possible Use Delphi technique Ensure size is estimated accurately Use a Code Counter to count logical SLOC size & differences Use a company Engineering Size & Productivity database Estimate probability points for uncertainty & risk Estimate at least 2 different ways at component level Use a checklist to ensure key estimation aspects are performed Perform an independent review looking at how good the estimate is in detail by staff who understand the software domain & estimation Develop feed-back loops between estimates and actuals Regularly re-estimate during s/w lifecycle for more accurate estimates 29
Any Questions? 30
BAE Systems Integrated System Technologies () Limited Victory Point Lyon Way, Frimley, Camberley Surrey, GU16 7EX United Kingdom Telephone +44 (0) 1276 603000 Fax +44 (0) 1276 603001 www.baesystems.com/insyte Steve Webb: email steve.a.webb@baesystems.com, Telephone 0208 329 5536 31