Software Technology Conference

Transcription

1 30 April 2003 Costing COTS Integration Software Technology Conference Salt Lake City Linda Brooks 1

2 Objective Provide a roadmap for doing an estimate for a Commercial Off-the-Shelf (COTS) software intensive system development Defining characteristics of COTS: Source code not available Future evolution of product not controlled by system developer 2

3 Topics Challenges of Estimating COTS Integration Components of an Estimate COTS Integration Software Estimating Methods Avoiding Common COTS Estimation Pitfalls Northrop Grumman Mission Systems Validation Example Summary 3

4 Topics Challenges of Estimating COTS Integration Components of an Estimate COTS Integration Software Estimating Methods Avoiding Common COTS Estimation Pitfalls Northrop Grumman Mission Systems Validation Example Summary 4

5 Challenges of Estimating COTS Sizing Productivity (size/effort) Limited availability of historical data and seasoned parametric models Non-separable effort in development Activities, life cycle phases and risks substantially different than new development COTS integration typically underestimated 5

6 COTS vs. Custom Analyze Design Implement Integrate Test Deliver Sustain Cost Factor * COTS Custom Marketing research Selection Analysis Design/Tailoring Implementation Interfaces/Integration Security Licenses Technical support Testing System Cut-Over Training Upgrades Replacement $$$ $$$ $ $ $$$??? $$$ $$ $$ $$$ $$ $$ $ $ $ $$$ $$$ $$??? $ $$ $$S $$ $$ $ $$ Most factors are project dependent * Cost of COTS versus Custom development needs to be evaluated across life-cycle Source: The MITRE Corporation. Used with permission. 6

7 Topics Challenges of Estimating COTS Integration Components of an Estimate COTS Integration Software Estimating Methods Avoiding Common COTS Estimation Pitfalls Northrop Grumman Mission Systems Validation Example Summary 7

8 Sample Cost Estimation Process Flow Plan the estimate Identify cost categories/ select estimating methods Estimate software size and cost drivers Generate cost estimate/risk analysis Perform sanity checks/ conduct reviews Prepare Basis of Estimates (BOEs) Review with higher level management/signoff Presentation will cover in more detail 8

9 Basic Cost Estimation Methods Direct comparison to historical data Adapted Comparison Cost Estimation Relationships (CERs) and standards Bottoms Up (engineering judgment based on detailed plan) Parametric cost model (preferably calibrated to org historical data) Level of effort (LOE) Quote for services or material to be purchased (COTS, Government-Off-the-Shelf [GOTS], NDI, etc.) 9

10 Estimate Software Size and Cost Drivers Identify and document software components to the lowest level possible Identify components expected to be satisfied by Developed software New software to be written Adapted Software Commercial off-the-shelf (COTS) software Includes COTS, GOTS, and NDI (Non-Developmental Items) where no source code is available Size each developed component and identify cost drivers* Identify COTS size and cost drivers* Identify size/cost driver ranges if appropriate (for risk analysis) * Can use definitions from selected parametric cost model 10

11 Risk Analysis Identify the the cost and risk drivers and quantify their impact Use cost categories, activities, cost driver settings, lessons learned from this briefing or other sources to trigger identification of risks Ideally develop at least three estimates (both cost and schedule) Best, worst, and expected baseline cases Monte-Carlo techniques can provide more sophisticated risk analysis 11

12 Components of a Complete Estimate Ground rules, assumptions Definitions & quantification of terms What the estimate includes (WBS Element, WBS Dictionary) Estimate of size, other cost drivers Required resources (labor hours) End products (Software, Specs, Manuals, etc.) Estimated schedule Estimated costs (labor and non-labor) Basis of Estimate (BOE) Risk Assessment 12

13 Topics Challenges of Estimating COTS Integration Components of an Estimate COTS Integration Software Estimating Methods Avoiding Common COTS Estimation Pitfalls Northrop Grumman Mission Systems Validation Example Summary 13

14 S/W Development Phase Activities: Custom & COTS REQUIREMENTS DESIGN IMPLEMENTATION ACCEPTANCE & DEPLOYMENT SUSTAINMENT Planning System Reqs Architecture Concept of Ops SW Reqs Custom Development Activities Preliminary Design Detailed Design Code & Unit Test Component Test System Test Acceptance Test Site Installation Site Activation Operations Maintenance Enhancements COTS Approach (Reqs Negotiation) COTS Component Identification COTS Assessment/Selection Prototyping COTS Specific Activities Glue Code Development Tailoring/Configuring COTS Integration & Test Component refresh Component Refresh Maintenance 14

15 COTS Software Integration Activities (1 of 2) Assessment (COCOTS) Research and evaluation of COTS products to select products to be used, 2 parts: Initial filtering of products Detailed assessment and evaluation Cognition (SEER-SEM) Understanding the functionality of the COTS software so as to have insight into, knowledge of, and comprehension of the receiving system/software in order to integrate it with the system or with other COTS components 15

16 COTS Software Integration Activities (2 of 2) Tailoring (COCOTS) Customization of the product for its intended use Provides required functionality missing in the COTS product, or Isolates users and/or developed software from unrequired/undesired capabilities or overly complex interfaces (e.g, wrappers ) Glue code (COCOTS) Developed software that Facilitates data or information exchange between the COTS component and the system or other COTS components into which it is being integrated Connects or hooks the COTS component into the system or to other COTS components but does not necessarily enable data exchange 16

17 COCOTS/COCOMO Costing View STAFFING Used with permission Chris Abts/Dr. Barry Boehm (USC) TIME 17

18 SEER-SEM COTS Costing View Summary level, define overall system characteristics Define overall program/csci characteristics, sums both custom and COTS components Custom software component New, reused, modified software SLOC, FP, Proxy sizing COTS glue code COTS Tailoring scripts, DDL, etc. COTS Software Component COTS functional sizing COTS specific cost drivers COTS Cognition License/Maintenance Fees PROGRAM SYSTEM PROGRAM CUSTOM COMPONENT UNIT UNIT PROGRAM COTS COMPONENT 18

19 SEER-SEM COTS Component Sizing IFPUG function point (FP) compatible sizing Quick Size - Computes sizes based on selection from COTS application list and % usage parameter Feature Sizing - FPs translated to terminology related to common COTS features Object Sizing FPs translated to terminology related to common COTS features 19

20 Candidate COTS Integration Estimation Methods Overall Method # Specific Method NEW ADAPTED 1 SEER-SEM b b b bc b bc bc bc b COCOMO II b b b 2 3 GLUE ASSESSMNT TAILORING COGNITION VOLATILITY INTEGRATION COCOTS b b b b b b Parametric Model b b b b b b Non-Model Estimate b b b b b SYSTEM I & T Custom Code COTS Custom/COTS C SEER COTS Unit b Normally Included b May Be Underestimated 20

21 Topics Challenges of Estimating COTS Integration Components of an Estimate COTS Integration Software Estimating Methods Avoiding Common COTS Estimation Pitfalls Northrop Grumman Mission Systems Validation Example Summary 21

22 COTS Tasks Typically Overlooked or Underestimated Systems and software engineering Hands-on prototyping Integrated system training and documentation Acquisition of COTS software in-depth knowledge -e.g., mentors, toolsmiths, vendor support Component and system performance tests Components and system regression tests with each upgrade Related software changes to support COTS software upgrades Glue code, database changes, configuration files Developer/operator training needed for COTS upgrades Source: The Aerospace Corporation. Used with permission 22

23 COTS Software Development Cost Estimation Method - 3 System Physical Architecture/Reqs WBS Estimated Labor Estimate Labor Per COTS Package Parametric Model Estimate COTS Tailoring(not in model est.) COTS Assessment COTS Volatility COTS Cognition Additional Integration (See Worksheet) Components New Code Adapted Code COTS Tailoring w SLOC/FP (e.g. Scripts, DDL ) Total Development Effort Estimate 23

24 COTS Worksheet (1 of 2): COTS Product Cost Drivers* COTS Product Descriptive Data: Product Name: COTS/GOTS/NDI: Version/Release Vendor Name Platform: (Manufacturer/Model/ OS) Functional Description: Product Functionality Utilization (%) COTS Product Characteristics Component Volatility Product Maturity Willingness to Extend Product Interface Complexity Product Support Training/Documentation Availability COTS Usage Parameters Component Selection Completeness Experience with Specific Component Experience with Similar Components Experience with Similar Component Experience with COTS Integration Integrator Personnel Continuity Learning Rate Technical Performance Constraints System Portability Architectural Engineering Component Integration and Test Test Level * Detailed definitions can be found in SEER-SEM manual or help screens 24

25 COTS Worksheet (2 of 2): COTS Integration Labor Estimation Sheet* A. COTS Assessment No. of products expected to be assessed: Avg. assessment hours per product: Total estimated assessment hours C. COTS Volatility No. of upgrades expected during development: Avg. hours per upgrade: Total estimated upgrade hours B. COTS Tailoring (effort not included with SLOC counts) No. of scripts expected: Avg. hours per script: No. of GUI developed screens expected: No. of hours per screen: Number of GUI developed reports expected: Avg hours per report No. of database tables expected to be defined Total No. of fields expected to be defined: Avg. hours per table Total estimated tailoring effort (hours) D. COTS Cognition Hours for COTS cognition consider parameters on previous page in developing estimate E. Additional COTS Integration Hours Additional hours for COTS integration not included in the other estimates Total labor hours to be added to software development effort for this COTS package (A+B+C+D+E): * More detailed components of A, B, and C can be found in Software Cost Estimation With COCOMO II. 25

26 Topics Challenges of Estimating COTS Integration Components of an Estimate COTS Integration Software Estimating Methods Avoiding Common COTS Estimation Pitfalls Northrop Grumman Mission Systems Validation Example Summary 26

27 Detailed code count Standard NGMS Process for Performing a Parametric Validation of Historical Data Identify COTS Products Identify Development Environment Parametric Model of Software Size, Effort, Schedule Map WBS to Common Model Identify Project Milestones Allocate Shared Support Functions Detailed Actual Cost, Effort by Subsystem, Phase, Function Source: K. Bradford, TRW, GSAW

28 Example of SEER-SEM Validation Successful calibration of SEER-SEM to Mission Systems COTS intensive completed project Used SEER COTS Components with SEER quick sizing to model COTS elements Used developed S/W components with SLOC counts for 4GL tailoring scripts and DDL Matched total actuals to within 7% of SEER estimate Included both COTS and non-cots development labor With adjustments came within 2% Provides confidence in method for future COTS estimates for similar COTS intensive developments 28

29 A calibrated model provides: Validation of provided data Basis of estimate for future enhancements Product % Diff A 12% B -9% C 31% D 4% E 20% F -43% G 0% H 6% Total 7% Source: K. Bradford, TRW, GSAW2001 Actual Vs Model of Delivered Software A B C D E F G H Product Product F was forced to restart with alternate COTS product 28% into the schedule. Actual Effort Model of Delivered Product Product C was severely constrained by staffing problems. Difference <2% with these corrections 29

30 Topics Challenges of Estimating COTS Integration Components of an Estimate COTS Integration Software Estimating Methods Avoiding Common COTS Estimation Pitfalls Northrop Grumman Mission Systems Validation Example Summary 30

31 Summary There are lessons learned from using and integrating COTS software applications, use them The necessary components of a complete estimate are always required, even with estimating for COTS usage Use cost drivers and sizing inputs from parametric models (such as SEER SEM or COCOTS) to derive your COTS integration estimate Cross check estimates by using results from two estimating methods Calibrate your estimating models and methodologies with historical data Evaluate the cost of COTS versus custom development across the life-cycle Identify the the cost and risk drivers and quantify their impact Revise estimates as you obtain more or better information 31

32 References SEER SEM Estimating Model, Galorath, Inc., Software Cost Estimation With COCOMO II, Barry Boehm et al, Prentice Hall, 2000 COCOTS Estimating Model, USC Center for Software Engineering, Kathy Bradford, Lessons Learned in Estimating the Software Cost of a Ground Station with COTS Integration, GSAW2001, 32

33 Acronym List BOM Bill of Materials CER Cost Estimating Relationship CM Configuration Management COCOMO COnstructive COst MOdel COCOTS COntstructive COTS COTS Commercial Off-the-Shelf DDL Data Definition Language FP Function Points GFE Government Furnished Equipment GOTS Government Off-the-Shelf GSAW Ground System Architectures Workshop GUI Graphical User Interface H/W Hardware LOE Level of Effort NDI Non-Developmental Item NGMS Northrop Grumman Mission Systems OCE Office of Cost Estimation QA Quality Assurance SEER-SEM System Evaluation and Estimation of Resources - Software Estimation Model S/W Software USC University of Southern California WBS Work Breakdown Structure 4GL Fourth Generation Language 33