Simple Empirical Software Effort Estimation Models

Size: px

Start display at page:

Download "Simple Empirical Software Effort Estimation Models"

Moris Nash
5 years ago
Views:

University of Southern California Center for

Barry Boehm, Ray Madachy 29th International

1 University of Southern California Center for Systems and Software Engineering Simple Empirical Software Effort Estimation Models Presenter: Brad Clark Co-Authors: Wilson Rosa, Barry Boehm, Ray Madachy 29th International Forum on COCOMO and Systems/Software Cost Modeling 21 October 2014

2 Topics Purpose Data Demographics Segmentation Strategy Analysis Results Conclusions and Next Steps 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 2

3 Purpose Statement Problem Defensible estimates are needed at the early conceptual phase of a software-intensive system s definition and development. A model with parameters is not very helpful if you don t have a defensible approach for specifying the inputs Proposed Solution Derive simple software effort estimation models for early lifecycle estimates Use data from DoD s repository of Software Resources Data Reports Examines the effect of size on effort by a segmentation strategy Size measured in terms of Equivalent Source Lines of Code Effort is represented in Person Months Segmentation strategy based on Software application domains System operating environments 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 3

4 Instrumentation Questionnaire: Software Resource Data Report (SRDR) (DD Form 2630) Source: Defense Cost Analysis Resource Center (DCARC) website: Content: Allows for the collection of project context, company information, requirements, product size, effort, schedule, and quality 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 4

5 Topics Purpose Data Demographics Segmentation Strategy Analysis Results Conclusions and Next Steps 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 5

6 Population and Sample Initial Dataset 474 records fully reviewed using GAO Best Practices 157 were excluded based on the following limitations: Inadequate information on reused and modified code Projects cancelled or terminated before delivery Inaccurate effort and schedule data Same duration (start and end dates) across software projects/components Missing effort or schedule data on more than 2 activities Duplicate records or submissions Estimates At Completion vice Actual Data Data reported at project level vice CSCI Final Dataset 317 projects included in the analysis 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 6

7 Number of Projects Operating Environments Satellite 18 Unmanned Aircraft Missile Ground Vehicle 42 Maritime Vessel 70 Aircraft Ground Fixed 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 7

Number of Projects Software Size 80 70 60 50 40 30 20 10 0 0-5 5-10 10-25 25-50 50-100

8 Number of Projects Software Size KESLOC 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 8

Number of Projects Age of Data 60 50 40 30 20 10 0 2004 2005 2006 2007 2008 2009 2010 2011 2012

9 Number of Projects Age of Data Delivery Year 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 9

10 Effort Distribution (%) Software Activities in SRDR Data Percent (%) Effort Distribution Software Requirements Analysis 10% Architecture / Detailed Design 15% Code and Unit Test 25% System / Software Integration 13% Qualification Test 9% Development Test & Evaluation 5% Other Direct Support 23% 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 10

11 Topics Purpose Data Demographics Segmentation Strategy Analysis Results Conclusions and Next Steps 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 11

12 Segmenting Data Segment software applications with similar characteristics, e.g. Product Reliability Product Complexity Reuse Availability Data Processing Platform Volatility Personnel Capability Personnel Skills Application Knowledge and Precedence Timing & Storage Constraints Development Environment These characteristics may not be known in early lifecycle estimation Estimation models based on data segmentation should be useful in early lifecycle estimation Assumption is that segments with similar characteristics will also have similar software development productivities The following three segmentation strategies are based on intuition and not empirical evidence (next slide) 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 12

13 Operating Environment Ground Site Ground Vehicle Maritime Vessel Aerial Vehicle Ordinance Vehicle Space Vehicle Fixed Mobile Segmentation Strategy 1 Acronym GSF GSM Examples Command Post, Ground Operations Center, Ground Terminal, Test Faculties Intelligence gathering stations mounted on vehicles, Mobile missile launcher Manned GVM Tanks, Howitzers, Personnel carrier Unmanned GVU Robots Manned MVM Aircraft carriers, destroyers, supply ships, submarines Unmanned MVU Mine hunting systems, Towed sonar array Manned AMV Fixed-wing aircraft, Helicopters Unmanned AVU Remotely piloted air vehicles Unmanned OVU Air-to-air missiles, Air-to-ground missiles, Smart bombs, Strategic missiles Manned SVM Passenger vehicle, Cargo vehicle, Space station Unmanned SVU Orbiting satellites (weather, communications), Exploratory space vehicles 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 13

Engineering Mission Support Automated Information Systems Super-Domains System Software Scientific & Simulation Process Control Test,

14 Strategies 2 & 3 17 Application Domains stratified into 4 general complexity zones called Super Domains Real-Time Signal Processing Vehicle Payload Vehicle Control Real Time Embedded Microcode & Firmware Command & Control Communication Software Configuration Item Engineering Mission Support Automated Information Systems Super-Domains System Software Scientific & Simulation Process Control Test, Mea, Diag, Equip. Software Tools Training Mission Planning Custom AIS Software Enterprise Service Sys Enterprise Info Sys Application Domains 14

15 Software Domain Survey -1 SEER-SEM Application Knowledge Bases (46) QSM Application Types (14) SRDR Application Domains (17) Microcode & Firmware Microcode & Firmware Signal Processing Signal Processing Vehicle Payload Flight Systems Avionics Vehicle Control Radar Embedded Electronics/Appliance Real Time Other Real Time Embedded Robotics Command/Control Command & Control Command & Control Communications Message Switching Network Management Telecommunications Communications Voice Provisioning Device Driver OS/Executive System Software System System & Device Utilities Process Control Process Control Process Control Artificial Intelligence Computer Aided Design (CAD) Expert System Mathematical & Complex Algorithms Scientific Scientific and Simulation Simulation Graphics Diagnostics Test, Measurement, and Diagnostic Equipment Super Domains (4) Real Time Engineering 15

16 SEER-SEM Application Knowledge Bases (46) Training / CBT / CAl Testing Software Software Development Tools Mission Planning & Analysis Electronic Data Exchange Graphical User Interface Multimedia Internet Server Applet Business Analysis Tool Business Intelligence Catalog/Inventory Management Customer Relationship Management Data Mining Data Warehousing Database Application Database System Decision Support System Document Management Financial Transactions Logistics Management Information Systems Office Automation Report Generation Transaction Processing Workflow Software Domain Survey -2 QSM Application Types (14) Business Business Financial Business Agile Business Government Business WEB Package Imp. (ERP) SRDR Application Domains (17) Training Software Tools Mission Planning Enterprise Service Systems Custom AIS Software Enterprise Information Systems Super Domains (4) Support Automated Information Systems 16

17 Topics Purpose Data Demographics Segmentation Strategy Analysis Results Conclusions and Next Steps 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 17

18 Modeling Approaches Three (3) approaches to modeling based on segmentation strategies 1. One model for each Operating Environment Application Domain pair 2. One model across all Operating Environments accounting for the Application Domain as a model factor 3. One model across all Operating Environments accounting for the Super- Domain as a model factor 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 18

19 Approach #1 One model for each Operating Environment Application Domain pair (Requires three model forms) PM = A KESLOC B PM = C + A KESLOC B PM = C + KESLOC B Variable Type Definition Person-Month (PM) Dependent Software engineering effort A Coefficient Multiplicative factor determined by data analysis Thousand Equivalent Source Lines of Code (KESLOC) Independent Size composed of new, modified, reused, and auto-generated counts B Coefficient Scaling factor determined by data analysis C Coefficient Fixed level of effort support activities (in Person-Months) 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 19

20 Application Domain Approach #1 Models One model for each Application Domain and each Operating Environment One model for all Domains and each Environment One model for each Domain and all Environments Operating Environment Ground Ground Space Maritime Aircraft Aircraft Ordinance Fixed Vehicle unmanned Vessel Manned Unmanned & Missile Total Software Tools Mission Planning Intel and Information Systems Scientific System Telecommunications Test Software Real-Time Embedded Mission Processing Vehicle Control Vehicle Payload Sensor Ctrl & Signal Processing Total Oct th International Forum on COCOMO and Systems/Software Cost Modeling 20

21 Approach #1: Effort Estimation Model Model for: Vehicle Control domain Ground Vehicle Manned environment R 2 MAD CV PRED KESLOC AD OE Model Form Obs. (%) (%) (%) (30) MIN MAX VC GVM PM = * KESLOC ^ % Tomorrow s presentation on the Software Cost Metrics Estimation Manual will show more of these models. 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 21

22 Approach #2 One model across all Operating Environments accounting for the Application Domain as a model factor Variable Type Definition Person-Month (PM) Dependent Variable Software engineering effort A Coefficient Multiplicative factor determined by data analysis Thousand Equivalent Source Lines of Code (KESLOC) Independent Variable Size composed of new, modified, reused, and auto-generated counts B Coefficient Scaling factor determined by data analysis Application Domains (AD) PM Independent Dummy Variables = A KESLOC B AD The treatment of the 12 application types (r) required the inclusion of 11 (r-1) dummy variables (D1,D2, D11) as denoted in next Slide 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 22

23 Approach #2 Dummy Variables Application Types Dummy Variables Software Tools TOOL D1 = 0 if TOOL Mission Planning MP D1 = 1 if MP, 0 if otherwise Information Systems AIS D2 = 1 if AIS, 0 if otherwise Scientific SCI D3 = 1 if SCI, 0 if otherwise System SYS D4 = 1 if SYS, 0 if otherwise Telecommunications TEL D5 = 1 if TEL, 0 if otherwise Test Software TST D6 = 1 if TST, 0 if otherwise Real Time Embedded RTE D7 = 1 if RTE, 0 if otherwise Command & Control CC D8 = 1 if MP, 0 if otherwise Vehicle Control VC D9 = 1 if VC, 0 if otherwise Vehicle Payload VP D10 = 1 if VP, 0 if otherwise Signal Processing SP D11 = 1 if SP, 0 if otherwise 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 23

24 Approach #2: Effort Estimation Model Equation: PM = KESLOC AD Where: PM = Engineering Labor in Person Months KESLOC = Thousand Equivalent Source Lines of Code AD = Application Domain multiplier Application Domain Multiplier Table AD Multiplier AD Multiplier TOOL 1.0 TST AIS RTE MP C&C SCI VC SYS VP TEL SP Oct th International Forum on COCOMO and Systems/Software Cost Modeling 24

Approach #2 Model Accuracy Measure Symbol Result Coefficient of Determination R 2 89% Coefficient of Determination (Cross-Validation Model) R 2 87% Coefficient of Variation CV 33% Measure of

25 Approach #2 Model Accuracy Measure Symbol Result Coefficient of Determination R 2 89% Coefficient of Determination (Cross-Validation Model) R 2 87% Coefficient of Variation CV 33% Measure of Magnitude F-test 214 P-Value α Sample Size N 317 Mean (Effort) Mean 431 Accuracy: Effect of KESLOC on effort is highly significant, when treated along with application domain Accuracy also examined using a cross-validation model with an R2 value of 87%. 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 25

Approach #3 One model across all Operating Environments accounting for the Super- Domain as a model factor Variable Type Definition Person-Month (PM) Thousand Equivalent Source Lines of Code (KESLOC)

26 Approach #3 One model across all Operating Environments accounting for the Super- Domain as a model factor Variable Type Definition Person-Month (PM) Thousand Equivalent Source Lines of Code (KESLOC) Super Domain (SD) PM Dependent Software engineering effort Independent Size composed of new, modified, reused, and auto-generated counts Independent Dummy Variables = A KESLOC B SD The treatment of the 4 Super Domains (r) required the inclusion of 3 (r-1) dummy variables (D1,D2, D3) as denoted in next Slide Super-Domains: Real-Time, Engineering, Mission Support & Automated Information Systems 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 26

Approach #3 Dummy Variables Super Domain Dummy Variables Mission Support MS D1 = 0 if MS Automated Information System AIS D1 = Engineering ENG D2 = Real Time RT

27 Approach #3 Dummy Variables Super Domain Dummy Variables Mission Support MS D1 = 0 if MS Automated Information System AIS D1 = Engineering ENG D2 = Real Time RT D3 = 1 if AIS, 0 if otherwise 1 if ENG, 0 if otherwise 1 if RT, 0 if otherwise 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 27

28 Approach #3: Effort Estimation Model Equation: PM = KESLOC SD Where: PM = Engineering Labor in Person Months KESLOC = Thousand Equivalent Source Lines of Code SD = Super Domain Multiplier Super Domain Multiplier Table Super Domain Multiplier N P-value t Stat Mission Support *** *** Automated Information System Engineering Real Time Oct th International Forum on COCOMO and Systems/Software Cost Modeling 28

Approach #3 Model Accuracy Measure Symbol Result Coefficient of Determination R 2 85% Coefficient of Variation CV 39% Measure of Magnitude F-test 446 P-Value α 0.

29 Approach #3 Model Accuracy Measure Symbol Result Coefficient of Determination R 2 85% Coefficient of Variation CV 39% Measure of Magnitude F-test 446 P-Value α Sample Size N 317 Mean (Effort) Mean 431 Accuracy: Effect of KESLOC on effort is highly significant, when treated along with application 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 29

30 Model Application and Limitations Application Useful for projects ranging between 1 and 842 KESLOC Limitation May be applicable to non-military sectors, to the extent that their practices are similar to those in the military. Although equation is not highly precise, it has the advantage of providing information on its relative accuracy. While controversy exists over whether or not SLOC is a good measure, consistent use of this metric provides for meaningful statistical results 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 30

31 Topics Purpose Data Demographics Segmentation Strategy Analysis Results Conclusions and Next Steps 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 31

32 Conclusion Simple effort estimation models appear feasible when used with a segmentation strategy Application Domains have a slightly better model fit to the data than Super- Domains, R 2 is 89% and 85% respectively Segmented models may provide more useful estimates at the early conceptual phase of a software-intensive system s definition and development where many common parametric model inputs are not known 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 32

33 Future Work Schedule estimation model considering Application Domain and Super-Domains Examine impact of other cost drivers Personnel capability Process maturity Requirements volatility Investigate alternative segmentation strategy based on productivity ranking: Rank all records in descending order of productivity Divide list into 10 groups Look for common cost drivers within each group (challenging) Effort Estimation Models using estimated software size (SLOC) Effort Estimation Model using Functional Requirements instead of SLOC 21 Oct th International Forum on COCOMO and Systems/Software Cost Modeling 33

Domain-Driven Software Cost, Schedule, and Phase Distribution Models:

Domain-Driven Software Cost, Schedule, and Phase Distribution Models: Using Software Resource Data Reports Wilson Rosa (DHS) Barry Boehm (USC) Brad Clark (SEI-CMU) Ray Madachy (NPS) Joseph P Dean (AFCAA)