HOW GOOD AN ESTIMATION PROCESS?

Transcription

1 1 HOW GOOD AN ESTIMATION PROCESS? Alain Abran Ecole de technologie supérieure University of Québec (Canada) ICEAA International Training Week October 17-20, 2016, Bristol (UK)

2 Alain Abran 20 years 20 years + 35 PhD Development Maintenance Process Improvement ISO: 19761, 9216, 25000, 15939, 14143, 19759

3 Agenda 3 1. Estimation: Craft or Engineering? 2. Estimation Phases Roles & Responsibilities 3. Conditions for credible estimation models 4. COSMIC Estimation models with data from industry 5. Conclusions

4

5 Mathematical Models Built with completed projects with almost no uncertainty in the inputs! 5

6 Inputs at Estimation Time 6 Imprecise Inputs at Feasibility Analysis Much Greater Error Range

7

8

9 9

10 10

11 11 Software Estimation Tools: Availability & Costs

12 Software Estimation Tools 12 Quality?

13 Consumers & Quality! 13

14 Estimation Tools & Quality? 14

15 (Software) Estimation Or? 15

16 Agenda Estimation: Craft or Engineering? 2. Estimation Phases Roles & Responsibilities 3. Conditions for credible estimation models 4. COSMIC Estimation models with data from industry 5. Conclusions

17 17

18 18

19 Mathematical Models Built with completed projects with almost no uncertainty in the inputs! 19

20 Inputs at Estimation Time 20 Imprecise Inputs at Feasibility Analysis Much Greater Error Range

21 21

22 22

23 23

24 24

25 Estimator role: Provide information about uncertainty range 25

26 Manager role: Pick a number & Manage Risk

27 Agenda Estimation: Craft or Engineering? 2. Estimation phases & Roles & responsibilites 3. Conditions for credible estimation models: A. Measurements of the Inputs B. Distribution of Data C. Quality of the Models 4. COSMIC Estimation models with data from industry 5. Conclusions

28 28

29 Issues with software metrics 29 1 st generation software size metrics Systematic errors: step function with min & max Invalid maths! No measurement unit Designed for unaccountability! Examples: - Usecase Points - Function Points - Story Points

30 30 Usecase Points & Similar Points-based Metrics Actors Dev. team Specs Usecases Programming language

31 31 Usecase Points & similar Points-based metrics Actor Table 1: Entities, Attributes, and Measurement Rules Entity Attribute Measurement rule Use case Specification of requirements Development team Programming language Complexity (of actor) Complexity (of use case) Relevance of the technical quality requirements Stability of the requirements Familiarity with the methodology Part-time status Analysis capability Application experience Object-oriented experience Motivation Difficulty The type of complexity (simple, average, or complex) of the interaction between the actor and the system The type of complexity (simple, average, or complex) measured in the number of transactions The level of relevance (from 0 to 5) of each of the 13 known non-functional qualities The level of stability (from 0 to 5) of the functional and nonfunctional requirements The level (from 0 to 5) of skills and knowledge of the development methodology in use for the project. The level (from 0 to 5) of part-time staff on the team The level (from 0 to 5) of analysis capabilities of the development team with respect to project needs The level (from 0 to 5) of team experience with the application domain of the system The level (from 0 to 5) of team experience with objectoriented design The level (from 0 to 5) of team motivation The level (from 0 to 5) of programming difficulty

32 32 Usecase Points & similar Points-based metrics Apples x Chairs Cars x x Books x Houses =?

33 33 Usecase Points & similar Points-based metrics Apples x Chairs x Books =? Cars x x Houses Primary School = Fail!

34 34 1 st Generation of Function Points: Step Functions! Function Points (FP) Key limitations: - Only 3 values - Limited ranges (min,max) - No single measurement unit of 1 FP! 6 FP 3 FP 4 FP 3-step size range for the IFPUG External Input Transactions Copyrights 2016: COSMIC and authors ICEAA Bristol (UK), Oct. 2016

35 2 nd Generation with COSMIC 35 COSMIC Function Points (CFP) No abitrary max A single CFP exists & is well defined Copyrights 2016: COSMIC and authors ICEAA Bristol (UK), Oct. 2016

36 1 st and 2 nd Generations of FSM 36 Function Points (FP) 3 FP 4 FP 6 FP COSMIC Function Points - CFP 2 1CFP No abitrary max A single CFP exists & well defined Copyrights 2016: COSMIC and authors ICEAA Bristol (UK), Oct. 2016

37 37 COSMIC sizes are measured on a true ratio scale There is no upper limit to the size of a functional process Largest observed functional processes? In avionics >100 CFP The size of the smallest change to an existing functional process is 1 CFP Open, freely available (via ) Copyrights 2016: COSMIC and authors ICEAA Bristol (UK), Oct. 2016

38 38 1 st & 2 nd generation of Function Points Methods 1 st generation ISO FSM Standard D FP s MkII FPA v nd generation MkII FPA Full FP s v.1 COSMIC FFP v. 2.0 COSMIC v Allan Albrecht FPA Feature Points IFPUG 4.0 IFPUG 4.1 IFPUG Copyrights 2016: COSMIC and authors ICEAA Bristol (UK), Oct. 2016

39 A sound Measurement Foundation 39 2 nd Generation Functional Size: No upper size limit No weigths No unsound mathematical operations Based on a concept common to all types of software of any size : a data movement A measurement unit: 1 data movement of a single data group A measurement symbol: 1 CFP COSMIC ISO (Common Software Measurement International Consortium)

40 Agenda Estimation: Craft or Engineering? 2. The estimation phases - Roles & responsibilites 3. Conditions for credible estimation models: A. Measurements of the Inputs B. Distribution of Data C. Quality of the Models 4. COSMIC Estimation models with data from industry 5. Conclusions

41 Another Key Issue: The quality of the inputs to the estimation models

42 Graph of the dataset 42 Copyright 2016 Alain Abran

43

44

45 Identification of outliers Copyright 2015 Alain Abran Outliers: Values of the data that are significantly far from the average of the population of the dataset. Candidate outliers: Typically at least 1 or 2 orders of magnitude larger than a data point closer to it. Can be identified from a graphical representation. The Grubbs test, also referred to as the ESD method (Extreme Studentized Deviate). The studentized values measure how many standard deviations each value is from the sample mean. When the P-value for the Grubbs test is less than 0.05, that value is a significant outlier at the 5.0% significance level. Values with a modified Z score greater than 3.5 in absolute value may well be outliers. 45

46 Frequency distribution 46 Size (Independent variable ); N=212 Effort (Dependent variable ); N=21 Copyright 2016 Alain Abran

47 Example of wrong decisions dues to outliers 47 A New Software Metric to Complement the IFPUG Function Points: The Software Non-functional Assessment Process: the SNAP Function Points

48 Author Very strong relationship of SNAP with Effort R 2 = 0,89 (R 2 max = 1,0) 48

49 49 Author s assertion on Figure 4: R 2 =.89 Spearman test for rank correlation of.85, with an associated confidence of statistical significance of greater than 99% (p-value <.0001).

50 50

51 Invalidity Range 51

52 What it really looks like for the range for which there is enough data points Approxmimatively: An R 2 = 0.3 Not R 2 = 0.89 (R 2 max = 1,0) 52

53 What it really looks like for the range for which there is enough data points Approxmimatively: An R 2 = 0.3 Not R 2 = 0.89 (R 2 max = 1,0) Conclusion: the claim that SNAP is useful for effort estimation is invalid 53

54 Log transformations Copyright 2015 Alain Abran Used when variables are not normally distributed Weak support for linear regression mathematical transformations The log transform is often used to obtain normal distributions for either the size or the effort variable, or both. 54

55 Scatter plot (n=312) 55 Copyrights 2016 Alain Abran

56 Agenda Estimation: Craft or Engineering? 2. The estimation phases - Roles & responsibilites 3. Conditions for credible estimation models: A. Measurements of the Inputs B. Distribution of Data C. Quality of the Models 4. COSMIC Estimation models with data from industry 5. Conclusions

57 Dataset with sparsely populated size intervals 57 Copyrights 2016 Alain Abran

58 Dataset with sparsely populated size intervals 58 Sparsely populated interval Copyrights 2016 Alain Abran

59 Dataset with sparsely populated size intervals 59 Copyrights 2016 Alain Abran

60 Representativeness & Population sample 60 = Outside of range! Copyrights 2016 Alain Abran

61 Agenda Estimation: Craft or Engineering? 2. Estimation phases & Roles & responsibilites 3. Conditions for credible estimation models: A. Measurements of the Inputs B. Distribution of Data C. Quality of the Models 4. COSMIC Estimation models with data from industry 5. Conclusions

62 62

63 COSMIC data from Industry 63 Practical experimentations with the COSMIC method in Automotive embedded software field By: Sophie Stern Renault

64 Renault Copyrights Renault 2016

65 Renault Copyrights Renault 2016

66 Renault Copyrights Renault 2016

67 67 Renault: Estimation & Negociations Copyrights Renault 2016

68 Renault COSMIC context & usage 68 Automated measurements Matlab Simulink 99% accuracy Estimation of CPU memory space based on COSMIC function points Planning of Requirements Specifications workload based on COSMIC functions points

69 Industry data with COSMIC 69 Productivity Analysis & benchmarking of projects from: Financial governmental organization (Canada)

70 70

71 71

72 72

73 73

74 74

75 Which estimation model to use in which contexts? 75

76 International benchmarking with ISBSG 76

77 Organization A: Effort = 31hrs/CFP x CFP + 2,411 hours ISBSG benchmark: Effort = 10hrs/CFP x CFP + 2,138 hours 77

78 Agenda Estimation: Craft or Engineering? 2. The phases in estimation 3. Economics concepts for estimation models 4. Estimation models with COSMIC: data from industry 5. Conclusions 1. Uncertainties range of candidate estimates 2. Conclusions

79 The inputs to productivity models have little uncertainty = Known Facts 79

80 Models Built with completed projects

81 Imprecise Inputs at Feasibility Analysis Much Greater Error Range 81

82 Recent Guidelines for Practitioners 82 A Guideline describing a range of Approximate Sizing methods Size/Cost estimates are usually needed before the FUR have been defined in detail A Guideline on Assuring the accuracy of COSMIC measurements The COSMIC Functional Size Measurement Method Version Guideline for Early or Rapid COSMIC Functional Size Measurement by using approximation approaches The COSMIC Functional Size Measurement Method Version Guideline for assuring the accuracy of measurements VERSION 0.93 July 2015 February 2011 Copyrights 2016: COSMIC and authors ICEAA Bristol (UK), Oct. 2016

83 Imprecise Inputs at Feasibility Analysis Much Greater Error Range The COSMIC Functional Size Measurement Method Version Guideline for Early or Rapid COSMIC Functional Size Measurement by using approximation approaches The COSMIC Functional Size Measurement Method Version Guideline for assuring the accuracy of measurements July 2015 VERSION 0.93 February

84

85 85

86 86

87 87

88 88

89

90

91

92

93 The Feel Good 93 ProjectCodeMeter assertions: A professional software tool to measure & estimate Time, Cost, Complexity, Quality & Maintainability of software projects, Development Team Productivity by analyzing their source code. Using a modern software sizing algorithm called Weighted Micro Function Points (WMFP) a successor to solid ancestor scientific methods as COCOMO, COSYSMO, Maintainability Index, Cyclomatic Complexity, & Halstead Complexity. More accurate results than traditional software sizing tools, while being faster & simpler to configure.

94 (Software) Estimation Or? 94

95 Building good estimation process & good estimation models 95 Sound Measurement Units Sound Maths! Recognition of uncertainties: how to recognize this & how to deal with it The estimator has to provide information, not a single estimate The manager has to select a single budget number & manage risks through contingency planning. Discipline, rigor, commitments & $$$

96 Want to know about good & bad practices in software estimation? 96

97 97