Just How Good is COCOMO and Parametric Estimation?

Transcription

1 Just How Good is COCOMO and Parametric Estimation? Tim Menzies (NC State) Barry Boehm (USC) Ye Yang (Stevens) Jairus Hihn (JPL) Naveen Lekkalapudi (WVU) October 21-23, th International Forum on COCOMO and Systems Software Cost Modeling All rights reserved.

2 Introduction The academic literature contains numerous ideas for novel cost estimation methods Yet even allowing for 20 years for maturation and infusion, few if any of these ideas have ever gained traction. (Mary Shaw, ICSE 2001) However, if you think about it parametric modeling has followed this pattern Why? Is it because professional cost estimators are stuck in old habits? Is it because most academics are Lost in a lost world? And not addressing real world issues The Moody Blues Is it a failure to communicate? Is it because parametrics just works? 2

3 Standard criticisms of parametric modeling Lines of code is stupid I know we all see it work but they refuse to listen Shepherd, Jorgensen, many SW engineers As programming becomes graphical then SLOC will make no sense Future is pointing towards integrating COTS and delivering services not developing code Too complicated to learn The mental model that underlies a parametric model is very different from the mental model of an engineer which is more analogy based Too expensive to implement Lots of variables need lots of data records Some calibration data is too old so need to be adding and culling data all of the time Some studies show that parametric estimation is a bad predictor due to sizing and other errors 3

4 Evolution of Methods Is it possible to model cost? What parameters & functional form? How deal with uncertainty? 1960s 1970s 1980s 1990s 2000s Today PERT, LSR Idea maturation Parametric Models Infusion Validated Robust Models Certification, Handbooks, Text Books Probabilistic Estimation Can we do cost and schedule? Cost and Schedule JCL How do we deal with sparse and noisy data? Draw line between 2 points Multivariate Regression Bayesian Analogy/Clustering 4

5 Back Ground The results presented are a part of a paper which has been submitted to ICSE Years of Parametric Effort estimation: A Report Card on COCOMO-style Research, Menzies, Boehm, et al. After June 2015 if you would like a copy contact me or go to (an adventure in itself) 5

6 Questions Addressed 1. Has parametric estimation been superseded by alternative methods? 2. Are the old parametric calibrations relevant to more recent projects? 6

7 Data Sources COCOMO II model inputs Rosetta stone used to convert from COCOMO 81 to COCOMO II 7

8 Methods Evaluated COCOMO II Out of the box COCOMO II COCONUT Calibrates COCOMO by randomly selecting points and searching area nearest Best values with random non local check. (thousands and thousand of times) Earlier paper compared to regression and found that gives same or better results measured by Median MRE CART Supervised clustering algorithm based on reducing prediction error of dependent variable can identify cluster drivers using a decision tree structure. Spectral clustering is unsupervised and cannot identify cluster drivers Knear Kth nearest neighbor algorithm 8

9 How evaluate methods Leave one out experiments Criteria is median (MRE) Magnitude of Relative Error MRE = abs(predicted Actual)/Actual 9

10 Questions Addressed 1. Has parametric estimation been superseded by alternative methods? Compare COCOMO II to Knear and CART 2. Are the old parametric calibrations relevant to more recent projects? Compare COCOMO II to COCONUT 10

11 Results 11

12 Results 12

13 Answers 1. Has parametric estimation been superseded by alternative methods? Answer - No COCOMO II either wins or ties 2. Are the old parametric calibrations relevant to more recent projects? Answer - COCOMO II calibration is relevant. COCONUT only shows a marginal improvement in performance. There are hints of some issues as there is a large variance for the COC05 data set 13

14 Conditions and biases associated with analysis All analysis is on COCOMO 81 or COCOMO II data Results support parametric modeling not just COCOMO Shows The original COCOMO II team did a good job. Evaluation based on actuals in the data sets so does not reflect human factors when estimates made in early lifecycle This is what Shepherd and others are often addressing Did not look at all model generation methods just a representative set 14

15 Last Word Combining yesterdays talk with todays the primary conclusion is If you have enough information use your favorite parametric model Early in the lifecycle use a proper analogy method like the one we are proposing Use analogy model to identify analogies for additional BOE data 15