BOE Training Material The Drive Toward data driven Estimating Boeing/Lockheed Martin/Northrop Grumman

BOE Training Material The Drive Toward data driven Estimating Boeing/Lockheed Martin/Northrop Grumman Original March 2011 Updated February 2013 1

Background/Objectives Background: There is a significant industry effort to increase the quality of contractor estimating processes. The industry has applied the term data driven estimating to this new method of developing and documenting our basis of estimates(boes). data driven Estimating methods have certain characteristics that are starting to emerge as standards within our industry. Objectives: What is driving our industry toward the use of data driven estimating methods and why is this a business imperative Review the current primary estimating methods Be able to identify the characteristics of good data driven estimating processes and how they are represented in our BOEs What are objective-scope-measures and how are they used in a data driven estimate Provide examples of data driven versus non-data driven BOEs 2

data driven Estimating - Contents BOE - Description data driven Estimating - Definition The Future Fundamentals of data driven Estimating Developing Relationships from History The Cost Estimating Relationship (CER) Process data driven Estimate Examples We are the Customer Summary Backup Charts/Supplemental Resources 3

Purpose of a BOE Documents and substantiates effort required to perform tasks Documents basis and assumptions Details understanding of requirements Conveys credibility by clearly defining tasks and assumptions Demonstrates that estimates are credible, defendable and realistic (historical data) Provides evidence that estimate is consistent with other proposal volumes Time-phased resources consistent with IMP/IMS Allocates skill mix appropriate for task Aligns with Work Breakdown Structure (WBS), SOW, and other product structures Meet customer funding, timing, and/or scheduling constraints 4

BOEs The Components A BOE consists of the following main elements: 1. Work Element ID WBS, SOW, Location, CLIN, NRE/Rec 2. Description of the products or services delivered and the tasks to be performed 3. Estimating method, rationale, assumptions, and computations used to develop the estimate Method of quoting 4. Relevant historical data and comparability of the historical data to the current program Explanation of any adjustments made to historical cost or programmatic data (i.e., use of modifiers) 5. Estimated labor resources to perform the defined tasks 6. Expenditure profile (time-phasing) of labor resources 7. Other elements as required by the cost proposal instructions 5

BOE Estimating Methods Labor estimates are developed using one or these typical estimating methods Judgment based on subject matter expert (SME) experience with similar efforts. Level-of-Effort/Support - uses resource-based estimates that are often based on a predetermined level of support for a given period of time Analogy/Comparison Method estimate based on historical frame of reference from a similar effort with adjustments for programmatic or technical differences Standard Time Estimating - Labor Standards and Realization/ Performance Factors - time necessary to complete a defined element of work following a prescribed (recognized) technique or method. Unit Method generally a simple ratio (not always statistically validated) Follow-on Based On Actuals extrapolation of actuals (CEAC, production) Parametric Cost Models use of calibrated, internally developed, or commercially available cost models Cost Estimating Relationships (CERs) - equations developed based on historical, statistically correlated relationships Note: Material or Other Direct Costs may be estimated by data such as quotes or catalogs. See FAR for specific cost element and degree of competition requirements 6

Customer Expectations Customers/DCAA/DCMA expect contractors to begin adopting improved estimating methods to support TINA-compliant proposals and other estimating products Least Preferred Judgment Most Preferred Fact Based Estimating Methods Move To Fact Based ASAP 7

BOE Methodologies Credibility Scale Customer Credibility High Low Labor Follow-on Projections based on Actuals Analogy/Comparison CER s Parametric Estimates / Cost Models Standards Unit Methods Level Of Effort Judgment Historical resource based estimates are easier to support than judgmental type estimates. 8

Customer Consequences Poor Estimating Failure to be responsive to customer estimating processes could be grounds for Non-compliance with RFP or government regulations Rejection of proposal and possibly defective pricing Estimating System Inadequacy Penalties for bad BOEs include: Questioned costs DCAA/Government Deficiency Report Negotiation losses Negative factor in competitive evaluations Defective pricing and fraud prosecution Suspension of bidding and debarment Holdbacks 9

data driven BOEs Basic Definition Fact Based Estimating: Estimating processes/boes that provide the supporting information and rationale necessary for accurate, credible and traceable estimates Information includes: Use of defined estimating methods Historical data from past programs Use of objective scope measures and complexity factors where appropriate Rationale addresses: Complete description of the estimating approach Explanation of the applicability of the historical information Estimating process must be documented, consistent and repeatable 10

Fact Based BOEs The Fundamentals 1. Estimate develop using a defined and documented estimating processes 2. Methods rely on scope measures (attributes) and complexity factors (parametrics) when applicable 3. Complete estimate development process is explained and documented in the BOE Professional judgments, modifiers/adjustment factors, and historical context is adequately explained Process is consistent and repeatable 4. Estimate relies on relevant historical data and context that is accessible and verifiable Shows use of latest and most current relevant data Supports all resource estimates with verifiable data 5. Estimating processes are monitored for learning and improvement as required to support future estimates 11

Objective Work-Scope Measures (OSM) Scope Measures: estimating predictive measures technical measures relating to the size of the effort (attributes) Generally physical and technical measures Sized at different product levels - system, subsystem, box, etc. Example: Work Products: # drawings by type, sheet counts, # documents & sheet count, SLOC by category, # requirements, # reviews, # test & type, weight, quantities, etc. Traits/Complexity Factors: characteristics that can improve the accuracy of relationships developed for the OSMs Example: Scope Measures = Drawings Traits: (numeric and descriptive): # Drawings, Type of Drawing; Drawing Heritage; Item Next Higher Assembly/Element; Number of Sheets; # of Revisions Example: Scope Measures = Source Lines of Code (SLOC) Traits: Software Language; New SLOC; Modified SLOC; Reuse SLOC; Actual Total Delivered SLOC, Deleted SLOC, Auto Generated Code, etc 12

Why Use Scope Measures and Traits in BOEs? Scope Measures - help to scope the size of the effort to be estimated Objective quantitative measures of scope to be performed Directly correlate to (predictors) resource estimates for work products and tasks Understanding the size (scope) of the proposed effort is critical to developing an accurate estimate (BOE) The BOE author should explain the derivation, source, and application of the scope measures in the BOE Traits/Complexity Modifiers address unique characteristics of the scope measures that can affect required effort to produce Capture the effects of technical and programmatic differences between historical programs and the current proposed effort. The produceability can vary widely for any scope measure - traits can be used to explain the differences Sometimes requires judgment 13

Objective Scope Measures - Examples Software Hardware Systems Engineering Candidate Estimating Attributes Source Lines of Code (SLOC) Function points Computer Software Configuration Item (CSCI) Number of drawings Power requirements Number/type of subsystems, subassemblies, components Volume (Dimensions) Weight Number/type of reviews Number of end items Number/type of tools Number of Contract Data Requirements Lists (CDRLs) Number of Subcontract Data Requirements Lists (SDRLs) Number of system requirements Number of Interface Control Drawings ( ICDs) Number of payloads Number of program segments Number of internal interfaces Number of external interfaces Number of risks Number of architectural trades 14

Traits/Complexity Factors - Examples Software Hardware Systems Engineering Estimating Modifier Design reuse Performance factors Complexity Reuse and pedigree New technology Components/functional complexity Schedule Requirements maturity Technology Readiness Level (TRL) requirements/maturity path Multi functional analyses Testing (qualification; prototype protoqualification; delta qualification; static; modal) Assembly, Integration and Test (AI&T)) Training requirements Mission profile and environments Mission design (data rates, pointing stabilities, Technical Performance Measurements (TPMs)) Oversight and analyses of subcontracted elements Risk Program maturity/life cycle Complexity Schedule Heritage Required documentation 15

Explaining Adjustment to a BOE In order to use an objective or subjective adjustment, an estimator must explain it thoroughly in the BOE. Provided below are some examples of adjustments that may be required to improve the accuracy of the estimate. The list is not meant to be totally inclusive of adjustments an estimator has available in their tool box. Technical Complexity ~ Labor Estimate Improvement Curves ~ Labor Estimate Schedule adjustments ~ Labor Estimate Quote Decrement ~ Material Estimate Quantity Adjustment ~ Material Estimate Escalation ~ Material Estimate Exchange Rate ~ Material Estimate 16

Using Historical Data in a BOE Use of Historical Data Must Address: Source of data generally from the organization s certified historical data repositories Complete identification of the nature, source and details of relevant historical data Ensure compliance with A.C.T.T.R. by documenting the following: Actual historical hours Charge numbers included Technical data used Time frame of historical data Repository or source of historical data 17

Developing a Relationship from Historical Data The statistical analysis process used in the development of a cost estimating relationship (CER) is documented in numerous industry and statistical sources The objective of the analysis is to validate that a significant statistical relationship exists for the CERs and to demonstrate that the CERs accurately predict effort or costs CER Strengths Past performance reflective of future costs Lean and credible estimating method in preparing BOEs More scientific and objective than judgement 18

Developing a data driven Relationship (CER) numerator denominator item to estimate known item pool base dependent variable independent variable Processs Step: 1. Identify Need 2. Establish Estimate Structure 3. Identification of Effort/Cost Drivers 4. Selection of Equation Form 5. Selection of Population 6. Data Collection 7. Data Normalization 8. Generate the Cost Estimating Relationship (CER) 9. Validate the Results 10. Document 19

Developing a data driven Relationship (CER) Regression Analysis: Develop mathematical relationship to make inferences about a future probabilistic event based on a sample of historical data. If relationship exist between two or more variables, such as weight and hours of effort/cost required, regression analysis is a powerful tool for the cost estimator/analyst. Regression analysis consists of curve fitting and the analysis of the descriptive and inferential statistics that result from the fit of the data. 20

Developing a data driven Relationship (CER) Y y b b x 0 1 Y y ax ( b 1) b Y b y ax ( 0 b 1) X Positive Linear Relationship X Positive Increasing Curve-linear Relationship X Positive Decreasing Curve-linear Relationship Y Y y b b x 0 1 y ax ( b 1) b X Negative Linear Relationship Negative Increasing X Curve-linear Relationship 21

Preparation of the Historical Data (CER) Step 1: Select the dependent variable to be estimated (e.g. results = hours, cost) Step 2: Determine the auditable source of historical data (both cost and technical measures) Step 3: Define the population(s), subpopulations, and sample(s) of data considering methods of collection (sites, programs, stratifications) Step 4: Select logical and measurable independent variables and determine their auditable sources of data (SLOCs, Drawings, Pages, Hours) Step 5: Extract the data and verify its accuracy and completeness using tests of reasonableness Step 6: Prepare the data for use in EXCEL or a statistical tool to perform the regression analysis Step 7: Test the resulting relationship for accuracy 22

Interpretation Decision Criteria of Statistical Results Statistical Measure Purpose of Measure Target Value for Good Fit Correlation Coefficient Coefficient of Variation Usability Criterion Measures the fit of the CER model. Gives a quick look at the percentage of the total variation that has been explained by the regression line. Measures the target goal for predictability within one standard error (68% of the data). The C.V. is calculated at the mean of Y, allowing the comparison of one CER to another. Calculated from the Standard error which measures the dispersion of the data. Judges reliability or usefulness of the CER measured on data points in analysis determining confidence of the results based on F-test when R 2 should be.80. R 2.80.50 < R 2 <.80.50 R 2 C.V. 50% 50% < C.V. < 100% C.V. 100% Population 10 6 Pop < 10 Pop < 6 Meets guidelines Below guidelines, acceptable with proper understanding Below guidelines 23

Data Driven Estimating - Examples We are all confronted in our daily lives with estimates for products and services E.g. Home remodel, auto repair We evaluate the credibility of these estimates based upon the content of the estimate We need to apply this customer mentality when we review our own proposal cost volumes and BOEs Example: Everyone has dealt with an automobile repair estimate that we struggle to feel comfortable with. Evaluate the following BOEs for automobile repair service how do we as consumers know that we are getting a fair price for these services 24

D&G Auto Poor, Fair or Good BOE Criteria Description of the products or services delivered and the tasks to be performed Estimated labor resources to perform the defined tasks Time-phasing of labor resources The estimating method, rationale, assumptions, and computations used to develop the estimate Identification of the relevant historical data and comparability of the historical program to the current program Explanation of any adjustments made to the historical data or estimate (i.e., use of modifiers) 25

data driven Estimating The Challenges Ahead Focus Improve estimating products across all estimating stakeholders (e.g. BOEs, Earned Value, Cost Analysis and Program Execution (CAPE), Independent Cost Evaluation (ICE), Business Case Analysis) The debate as we evolve to our future At what level will we be generating estimates (e.g. subsystem, assembly) What scope measures (OSM) will we be using How will we document our estimating processes Same or different approaches for firm business, ROMs/NTE, EAC/ETC? What measures/approaches will be cost credible with our customers Historical databases, processes and strategies need to be flexible as we evolve toward the future improved estimating products Estimating, Cost Modeling and Engineering teams working together As the desired characteristics of these high quality estimating methods evolve our methods must evolve also. Immediate needs: Documented estimating process Good historical data Libraries of BOEs demonstrating data driven estimating approaches 28

Backup Charts/Supplemental Resources 29

Estimating Method - Judgment Judgment: Least preferred, least credible estimating method very subject to bias Generally not based upon historical data, scope measures or other sound estimating approaches Relies on Subject Matter Expert (SME) estimate s of small work packages aggregated to complete hardware, software, etc tasks State author s experience from which to make an estimate Programs worked; Years in industry Decompose effort into well defined tasks and subtasks Disclose if historical methods were investigated and not used, and why they were not used Rarely are we asked to quote something that does not have some similar historical likeness! 30

Estimating Method - LOE Level of Effort (LOE) - Use when customer specifies effort to be provide over a stated period of time on work that can be stated only in general terms Requirement for specific skills/talents that tend to be headcount driven Used when product cannot be well defined Contract is for delivery of hours not product. Payment based on effort expended not results achieved. Restricted LOE Contracts Statement of Work (SOW) that specifically calls out LOE by customer What is NOT LOE Level loading of labor not specifically directed by customer Staffing a position (i.e. Program Manager) as 160 hrs/month during period of performance on a completion type contract 31

Estimating Method Analogy/Comparison Analogy/Comparison: This method is based on historical context or frame of reference from a similar effort. Compares items being estimated to items of similar configuration and known cost to produce estimate Includes use of historical data Addresses differences between historical data and proposed solution being estimated May use relevant estimating attributes and complexity or adjustment factors (modifiers) to reflect programmatic, technology, requirements or technical differences Clearly document in BOE why and how factors are used and validated Estimated Cost (New Design) = Historical Cost (Similar Design) + Complexity or Scaling Factor 32

Estimating Method Standard Time Labor standard - time necessary to complete a defined element of work following a prescribed method, for an average (qualified) worker working at a 100% efficiency with no hindrances, breaks, delays or fatigue over extended periods of time Realization/Performance factors recognize actual experience measured against the standards This method is most often used in manufacturing or production related estimates. Examples: Manufacturing operation: Time required to complete a drilling operation on a product adapter Car servicing: Factory maintenance manual sets a standard number of minutes for each type of repair 33

Estimating Method Unit Method Unit Method: Relies on an accumulation of past experience divided by cost driver to produce cost per unit Example: Labor cost of developing and fielding entire ground station and costing using number of ground stations to be fabricated Example: ABC widget has 11 square feet of surface area. Actual design hours for XYZ widget (10 square feet of surface area) were 80 hours. 80 hours/10 square feet = 8 hours per square foot. Therefore ABC widget will take 88 (8 X 11) hours to design A Simple Ratio (not statistically validated) of X to Y Is A Unit Method 34

Estimating Method Follow-on Effort (Actuals) Follow-on based upon program actuals Commonly used when developing an estimate of labor or material for continuing efforts (usually) Production hardware Spare parts, Repairs, or Retrofit Sustaining Engineering Estimate At Complete (EAC) Definition: An estimate that uses historical data (hours, material dollars, weights, etc.) to project continuing future efforts of the same product line, process or part. 35

Estimating Method Parametric Model Parametric Model: Utilizes mathematical calibrated models that derive project cost or effort from attributes / metrics of project Source lines of code (SLOC), function points (FPs), COTS product characteristics, Weight, Size The measurement is based on the technical, physical, or other end item characteristics Facilitates alignment of program baselines (e.g. IMS and Staffing plans) Customer RFP often dictates use Utilized by customer to evaluate BOEs and bid overall Used by ICE to evaluate bid position & performance risk Performed by an experienced cost modeler Commercial parametric models include: PRICE, SEER 36

Estimating Method Parametric Model, Cont Parametric Model: is a group of cost estimating relationships used together to estimate entire cost proposals or significant portions thereof. These models are often computerized and may include many interrelated CER s both cost-to-cost and cost-to-non-cost estimating techniques estimating relationships to predict complex Some models use a very limited number of independent estimated values and a series of Parametric inter-related cost-to-noncost estimating techniques relationships to predict complex proposal cost structures 37

Estimating Method - CER Cost Estimating Relationships: Statistical estimating methods that produce mathematically fitted functions Developed by mathematically relating effort or time estimates to OSM, traits or other feature of product or manufacturing environment Relationship may be based on single or multiple measures (independent variables) representing estimating attributes, complexity measures, or other measures of effort. The results from application of the CERs (dependent variable) is frequently a labor estimate expressed in hours/staff-months or an estimate of time in days/weeks/months. Example: Based on actuals of four projects (design hrs/ sq ft) using linear regression, this estimating relationship is created: Design Hours = 9.32 * sq ft - 14.3 38

Hardware Cost Modeling Estimate system acquisition costs based upon: Quantitative parameters such as complexity, quantity, weight, and size; and Quantitative parameters such as environmental specification, type of packaging, and level of integration; and schedule parameters such as months to first prototype, manufacturing rate, and amount of new design Electronic hardware items are characterized by the electronic application and type of componentry Mechanical and structural elements can be described in terms of their construction, type of material, functionality, machinability and manufacturing process 39

data driven Estimating Why Basis of Estimate (BOE) Focus? BOE - Description data driven Estimating - Definition Fundamentals of data driven Estimating data driven Estimate Examples We are the Customer Non-Proprietary BOE Training Material Available to All 40

Background/Objectives Background: There is a significant industry effort to increase the quality of contractor estimating processes. The industry has applied the term data driven estimating to this new method of developing and documenting our estimates/boes. data driven Estimating methods have certain characteristics that are starting to emerge as standards within our industry. Objectives: What is driving our industry toward the use of data driven estimating methods and why is this a business imperative Be able to identify the characteristics of good data driven estimating processes and how they are represented in our BOEs Provide examples of data driven versus non-data driven BOEs Raise the bar by providing access to non-proprietary BOE training 41

Mapping of DFARS 252.215.7002 Cost Estimating System Requirements to Major Estimating System Components 13. Provide for internal review of, and accountability for, the acceptability of the estimating system, including the budgetary data supporting indirect cost estimates and comparisons of projected results to actual results, and an analysis of any differences. Roles 1. Establish clear responsibility for preparation, review, and approval of cost estimates and budgets. 2. Provide a written description of the organization and duties of the personnel responsible for preparing, reviewing, and approving cost estimates and budgets. 12. Require management review, including verification of the estimating and budgeting policies, procedures, and practices. Internal Controls Responsibilities 3. Ensure that relevant personnel have sufficient training, experience, and guidance to perform estimating and budgeting tasks in accordance with the established procedures. Estimate Approval 5. Provide for adequate supervision throughout the estimating and budgeting process. 6. Provide for consistent application of estimating and budgeting techniques. 7. Provide for detection and timely correction of errors. 8. Protect against cost duplication and omissions. 10. Require use of appropriate analytical methods. 14. Provide procedures to update cost estimates and notify the Contracting Officer in a timely manner throughout the negotiation process. 16. Provide estimating and budgeting practices that consistently generate sound proposals that are compliant with the provisions of the solicitation and are adequate to serve as basis to reach a fair and reasonable price. 17. Have an adequate system description, including policies, procedures, and estimating and budgeting practices, that comply with the Federal Acquisition Regulation and Defense Federal Acquisition Regulation Supplement. Estimate Review Estimate Preparation Estimate Data Sources Training 4. Identify and document the sources of data and the estimating methods and rationale used in developing cost estimates and budgets. 9. Provide for the use of historical experience, including historical vendor pricing information, where appropriate. 11. Integrate information available from other management systems. 15. Provide procedures that ensure subcontract prices are reasonable based on a documented review and analysis provided with the prime proposal, when practicable. 4 Well Supported Data Driven Estimating in Key to Estimating System Health 42

DFARS 252.215.7002 Cost Estimating System Requirements 4. Identify and document the sources of data and the estimating methods and rationale used in developing cost estimates and budgets. 9. Provide for the use of historical experience, including historical vendor pricing information, where appropriate. 11. Integrate information available from other management systems. 6. Provide for consistent application of estimating and budgeting techniques. 10. Require use of appropriate analytical methods. 15. Provide procedures that ensure subcontract prices are reasonable based on a documented review and analysis provided with the prime proposal, when practicable. Provide for adequate supervision throughout the estimating and budgeting process. Estimate Data Sources Estimate Preparation Estimate Review 43

Purpose of a BOE Documents and substantiates effort required to perform tasks Documents basis and assumptions Details understanding of requirements Conveys credibility by clearly defining tasks and assumptions Demonstrates that estimates are credible, defendable and realistic (historical data) Provides evidence that estimate is consistent with other proposal volumes Time-phased resources consistent with IMP/IMS Allocates resources such as skill mix appropriate for task Aligns with Work Breakdown Structure (WBS), SOW, and other product structures Meet customer funding, timing, and/or scheduling constraints 44

BOEs The Components A BOE consists of the following main elements: 1. Work Element ID WBS, SOW, Location, CLIN, NRE/Rec 2. Description of the products or services delivered and the tasks to be performed 3. Estimating method, rationale, assumptions, and computations used to develop the estimate Method of quoting 4. Relevant historical data and comparability of the historical data to the current program Explanation of any adjustments made to historical cost or programmatic data (i.e., use of modifiers) 5. Estimated labor and ODC resources to perform the defined tasks 6. Expenditure profile (time-phasing) of resources 7. Other elements as required by the cost proposal instructions 45

BOE Estimating Methods Estimates are developed using one or more of these typical estimating methods Judgment based on subject matter expert (SME) experience with similar efforts. Level-of-Effort/Support - uses resource-based estimates that are often based on a predetermined level of support for a given period of time Analogy/Comparison Method estimate based on historical frame of reference from a similar effort with adjustments for programmatic or technical differences Standard Time Estimating - Labor Standards and Realization/ Performance Factors - time necessary to complete a defined element of work following a prescribed (recognized) technique or method. Unit Method generally a simple ratio (not always statistically validated) Follow-on Based On Actuals extrapolation of actuals (CEAC, production) Parametric Cost Models use of calibrated, internally developed, or commercially available cost models Cost Estimating Relationships (CERs) - equations developed based on historical, statistically correlated relationships (See full training package for more details) 46

Customer Consequences Poor Estimating Failure to be responsive to customer estimating processes could be grounds for Non-compliance with RFP or government regulations Rejection of proposal and possibly defective pricing Holdbacks Penalties for bad BOEs include: Questioned costs DCAA/Government Flash Report Negotiation losses Negative factor in competitive evaluations Defective pricing and fraud prosecution Suspension of bidding and debarment 49

Fact Based BOEs The Fundamentals 1. Estimate developed using a defined and documented estimating processes 2. Methods rely on scope measures (attributes) and complexity factors (parametrics) when applicable 3. Complete estimate development process is explained and documented in the BOE Professional judgments, modifiers/adjustment factors, and historical context is adequately explained Process is consistent and repeatable 4. Estimate relies on relevant historical data and context that is accessible and verifiable Shows use of latest and most current relevant data Supports all resource estimates with verifiable data 5. Estimating processes are monitored for learning and improvement as required to support future estimates 51

Objective Work-Scope Measures (OSM) Scope Measures = estimating predictive measures technical measures relating to the size of the effort (attributes) Generally physical and technical measures Sized at different product levels - system, subsystem, box, etc. Example: Work Products: # drawings by type, sheet counts, # documents & sheet count, SLOC by category, # requirements, # reviews, # test & type, weight, quantities, etc. Traits/Complexity Factors: characteristics that can improve the accuracy of relationships developed for the OSMs Example: Scope Measures = Drawings Traits: (numeric and descriptive): # Drawings, Type of Drawing; Drawing Heritage; Item Next Higher Assembly/Element; Number of Sheets; # of Revisions Example: Scope Measures = Source Lines of Code (SLOC) Traits: Software Language; New SLOC; Modified SLOC; Reuse SLOC; Actual Total Delivered SLOC, Deleted SLOC, Auto Generated Code, etc 52

Explaining Adjustment to a BOE In order to use an objective or subjective adjustment, an estimator must explain it thoroughly. Provided below are some examples of adjustments that may be required to improve the accuracy of the estimate. The list is not meant to be totally inclusive of adjustments an estimator has available in their tool box. Technical Complexity ~ Labor Estimate Improvement Curves ~ Labor Estimate Schedule adjustments ~ Labor Estimate Quote Decrement ~ Material Estimate Quantity Adjustment ~ Material Estimate Escalation ~ Material Estimate Exchange Rate ~ Material Estimate 54

data driven Estimating - Examples We are all confronted in our daily lives with estimates for products and services E.g. Home remodel, auto repair We evaluate the credibility of these estimates based upon the content of the estimate We need to apply this customer mentality when we review our own proposal cost volumes and BOEs Example: Everyone has dealt with an automobile repair estimate that we struggle to feel comfortable with. Evaluate the following BOEs for automobile repair service how do we as consumers know that we are getting a fair price for these services 56

data driven Estimating The Challenges Ahead Focus Improve estimating products across all estimating stakeholders (e.g. BOEs, EV, CAPE, ICE, Business Case Analysis) The debate as we evolve to our future At what level will we be generating estimates (e.g. subsystem, assembly) How will we document our estimating processes Historical databases, processes and strategies need to be flexible as we evolve toward the future improved estimating products Estimating, Cost Modeling and Engineering teams working together As the desired characteristics of these high quality estimating methods evolve our methods must evolve also. Immediate needs: Documented estimating process Good historical data Libraries of BOEs demonstrating data driven estimating approaches 60

Back Up 61

Estimating Method - Judgment Judgment: Least preferred, least credible estimating method very subject to bias Generally not based upon historical data, scope measures or other sound estimating approaches Relies on SME estimate s of small work packages aggregated to complete hardware, software, etc tasks State author s experience from which to make an estimate Programs worked; Years in industry Decompose effort into well defined tasks and subtasks Disclose if historical methods were investigated and not used, and why they were not used Rarely are we asked to quote something that does not have some similar historical likeness! 63

Estimating Method - LOE Level of Effort - Use when customer specifies effort to be provide over a stated period of time on work that can be stated only in general terms Requirement for specific skills/talents that tend to be headcount driven Used when product cannot be well defined Contract is for delivery of hours not product. Payment based on effort expended not results achieved. Restricted LOE Contracts SOW that specifically calls out LOE by customer What is NOT LOE Level loading of labor not specifically directed by customer Staffing a position (i.e. Program Manager) as 155 hrs/month during period of performance on a completion type contract 64