Effective Software Sizing A Galorath Web Seminar 01 September 2004

Transcription

1 Effective Software Sizing A Galorath Web Seminar 01 September

2 Effective Software Sizing What are we estimating? What is software? Why do we care? What do we mean by software size? How do we measure it? What are the units? Lines of code Function Points Other measures Total vs. Effective Size Size Estimate Growth Size Estimate Uncertainty How do we estimate? Direct measurement Function points Requirements Other design artifacts Code Counts (pre-existing) Historical comparison methods Single analogy Database analysis Delphi Methods Structured comparisons (AccuScope) Developing a high-fidelity size estimate Galorath process 2

3 How Big is My Software Project? 50,000 what? 50,000 according to whom? Why not 30,000? 50,000 plus or minus what? 50,000 with or without requirements creep? 3

4 Why We re Concerned about Software Size: The Size Effort Relationship Development effort varies in direct proportion to the size of a program Knowing size allows estimators to determine effort (cost) Better size estimates = better cost estimates Software Size is the main driver of software development cost, effort and schedule -- use the best available estimate of size range! Development Schedule Months Size (SLOC) Development Effort Months Size (SLOC) 4

5 Galorath Observations from Reviewing Program Estimates Sizing Mistake Not enough time/effort spent on software sizing in general No clear definition of size, or lack of common understanding of definition Size as an Independent Variable technique changing the size estimate to get the desired cost or schedule estimate, without considering impact on functionality Historical actuals ignored as analogies due to differences in language and methodology Consequence Unbelievable estimates results don t match the program and are too optimistic or pessimistic Inconsistent estimates results don t pass the sanity check, unreliable output, blame the model Unrealistic size (and effort and schedule) estimates, unexecutable projects Lost opportunity to better forecast future from past 5

6 What is it you re trying to estimate? The most critical step in estimating anything is to first understand what it is. Peter Korda 6

7 What Do We Mean by Software Size? Size (work units) Desire Software Development Process Software Size (work units) Viewpoint: Essence (the what) Viewpoint: Implementation (the how) 7

8 Effective Software Sizing What are we estimating? What is software? Why do we care? What do we mean by software size? How do we measure it? What are the units? Lines of code Function Points Other measures Total vs. Effective Size Size Estimate Growth Size Estimate Uncertainty How do we estimate? Direct measurement Function points Requirements Other design artifacts Code Counts (pre-existing) Historical comparison methods Single analogy Database analysis Delphi Methods Structured comparisons (AccuScope) Developing a high-fidelity size estimate Galorath process 8

9 Common Size Metrics Functional Function Points Requirements Design Objects Use Cases Technical Source Lines of Code Executable Lines of Code 9

10 Which Size Metric at Which Phase? Concept Requirements Design Completion Analogies Let you size items when size is still a fuzzy concept. Can translate directly into lines Functional Metrics Map functionallyoriented requirements documents directly into a sizable basis whether lines or functions These also may be appropriate at an earlier stage, depending on project and design circumstances Design Metrics As the software assumes a form, seek a metric that mirrors the developers perspective, such as OO metrics, lines, effective lines Lines or similar Grammatical Metrics Unarguably, a bedrock metric if obtainable, they can be counted precisely. At this point, function based size can be counted within 10% 10

11 Total vs. Effective Size Size (work units) Desire Software Development Process Software Size (work units) Viewpoint: Essence (the what) Total software size reflects the total functionality of the program Viewpoint: Implementation (the how) Effective Size is more directly related to the amount of effort required to develop the software, considering Redesign, reimplementation and/or retesting of pre-existing code Use of existing design or concept Use of code generators, GUI Builders, etc. 11

12 Total vs. Effective Size Many programs include a combination of new and pre-existing software New code needs to be designed, coded, tested, integrated Pre-existing code May require some degree of design change, recoding, retesting, reintegration May also require reverse engineering to discover how it works Misconceptions about reuse will impact the estimate 12

13 Total vs. Effective Size Size (Total SLOC) Change in size mix may have no impact on functionality Total Size Requirements Design Coding Size (SLOC) Phase Total Size Total Size Requirements Design Coding Phase No change in Total Size Pre- Existing Lines of Code New Lines of Code 13

14 Total vs. Effective Size Total vs. Effective Size Growth Effective Size grows 54% Size Effort and Schedule Growth 0 Requirements Design Coding Total Size Phase Effort Estimate +61% Schedule Estimate +27% No additional functionality Effective 80 Size Requirements Design Coding Phase Effort (Thousands of Hours) Schedule (Months) 14

15 Total Size Growth Source: In addition to effective size growth, the total size estimate tends to grow over time Some sharp growth periods due to addition of functionality, better understanding of requirements or design changes Steady growth from approximately development midpoint through acceptance testing can occur due to response to trouble reports Some size growth can also be attributed to institutional optimism Bottom Line: You must understand size, and factor code growth in your estimates, or your effort & schedule projections probably will be low 15

16 Expressing Uncertainty The actual, final size of the finished program is a point value. However Estimates of size and difficulty expressed as single point values don t tell the whole story: How confident am I in this value; i.e., what is the probability of not exceeding this value? How certain am I in this value; i.e., how wide is the probability distribution? Three-point estimates are better: LEAST: 1% Probability; I can t imagine the result being any smaller than this. LIKELY: Best Guess; If I were forced to pick one value, this would be it. MOST: 99% Probability; I can t imagine the result being any larger than this. 16

17 Effective Software Sizing What are we estimating? What is software? Why do we care? What do we mean by software size? How do we measure it? What are the units? Lines of code Function Points Other measures Total vs. Effective Size Size Estimate Growth Size Estimate Uncertainty How do we estimate? Direct measurement Function points Requirements Other design artifacts Code Counts (pre-existing) Historical comparison methods Single analogy Database analysis Delphi Methods Structured comparisons (AccuScope) Developing a high-fidelity size estimate Galorath process 17

18 Measurement or Estimation? Some software size metrics can be directly measured at certain stages of development (but may be estimated at any stage) Function Points based on detailed requirements Objects or Use Cases based on design artifacts Requirements based on structured requirements documents Lines of Code (pre-existing software) Other metrics or stages require estimation Lines of Code (before they re written) Function Points (prior to detailed requirements) Object-oriented or use case metrics (prior to design) 18

19 Measured Metrics Count key elements of the available abstraction(s) Business Requirements documented during a Goals and Objectives sessions Actors and Use Cases that form a Unified Modeling Language (UML) Use Case Diagram Structured Text requirements Function Points Programs Modules Source Lines of Code (SLOC) if the code already exists 19

20 Estimated Metrics Virtually any metric which can be measured can also be estimated Fortunate, since effort and schedule estimates are frequently required before the requisite information is available to be measured Estimated metrics can include: Source Lines of Code (either total or effective) Function Points Requirements Actors / Use Cases 20

21 Estimation Techniques Expert Judgment What does the development team think? Delphi Techniques Structured process for collecting and distilling knowledge from a group of experts Analogies Compare to a program with known size and similar functionality Database Comparison Evaluate size range for a number of similar programs Structured Pairwise Comparison Analytical Hierarchy technique incorporated in SEER-AccuScope 21

22 Expert Judgment Advantages: Can be very accurate and effective, especially if the expert has a great deal of domain knowledge and experience If the expert is part of the development organization, the project team is more likely to buy into the size estimate and resulting effort/schedule estimate Specific historical data is not required Challenges: True experts may be hard to find New or different types of application may be difficult to estimate accurately Hard to tell the difference between a solid expert judgment and a SWAG 22

23 Delphi Methods Delphi techniques use a panel of knowledgeable persons, responding anonymously to a series of questions. Each panelist is provided with a summary of opinions before each series of questions. In each series, it is expected that the range of estimates will decrease and the median will approach the correct answer Advantages: Incorporates a range of expertise, not just one individual Results in a statistically derived range with a known confidence level Specific historical data is not required Challenges: Need to assemble a panel of experts Additional time, effort and coordination required to conduct the study 23

24 Analogy Methods Specific program for which actual size information is available is selected, and the program to be estimated is compared. Judgements are made as to whether the unknown program is bigger, smaller or about the same as the analogous program. Advantages: Based on specific historical data Takes advantage of familiarity with previous projects Challenges: Specific historical data must be available Normalization of metrics may be required Depends on a thorough understanding of the similarities between the programs 24

25 When is an analogy an analogy? Data points that are size analogous have similar: Platform Application Acquisition method Development method Development standard Language Functionality Implementation date More characteristics in common is better, although a data point should not necessarily be thrown out if it lacks a particular common characteristic Often the estimator will need to use judgment as to whether a data point is analogous or not. Be sure to choose analogies that you can explain why it is an analogy! 25

26 Maintaining Analogy Data Points Match proposed products, features, and characteristics to products, features, and characteristics of known size Mutually exclusive choices Combinations of choices Deviation from choice(s) Maintain products, features, and characteristics lists with associated actual sizes in a history repository SEER-AccuScope provides this capability 26

27 Database Analysis A sizing database can be an extremely powerful tool for using past data to predict the future. Data points which are similar to the unknown program are selected from the database, and a statistical analysis of the range and median of those points is conducted. Advantages: Incorporates more data to achieve better statistical significance Results in a statistically derived range with a known confidence level Outstanding sanity check on other methods Challenges: Significant effort may be required to develop and maintain a high-quality software size database Normalization of size metrics may be required Selection of appropriate data points for comparison is important May produce wide ranges 27

28 Database Analysis Size (normalized SLOC) Data Point + Std Dev Median - Std Dev 28

29 Maintaining Historical Data SEER Repository Maintains historical data in SQL database format Integrates with SEER-SEM and SEER-AccuScope SEER-SEM and SEER-AccuScope handle normalization of different metrics 29

30 Sizing by Pairwise Comparison Compare each and every possible couple combination Analyze logical consistency of comparison results Compute relative magnitudes of each element Bind unknown elements to an absolute size 30

31 Which Technique to Use? Depends on: Available expertise Available data Do we have information about analogous programs? Do we have access to a size database? Available time Hunting for experts Hunting for data More fidelity = more time and effort to estimate size Requirement for fidelity Is this a ROM planning estimate, a budgetary estimate, a contractual estimate? 31

32 There is no single best method for estimating software size Any method for assessing software size for any given development project will have advantages and disadvantages Since this is estimation, no single correct answer is possible Always consider size estimates as a range which reflects the level of confidence A comprehensive, organized sizing study which considers multiple methods can be used to derive a high-fidelity size estimate 32

33 Effective Software Sizing What are we estimating? What is software? Why do we care? What do we mean by software size? How do we measure it? What are the units? Lines of code Function Points Other measures Total vs. Effective Size Size Estimate Growth Size Estimate Uncertainty How do we estimate? Direct measurement Function points Requirements Other design artifacts Code Counts (pre-existing) Historical comparison methods Single analogy Database analysis Delphi Methods Structured comparisons (AccuScope) Developing a high-fidelity size estimate Galorath process 33

34 Size Study Methodology Evaluate All Sources of Software Size New Size Preexisting Size (rework) Generated Code COTS/GOTS Glue Code Integrated Code Total Size Estimates Least Likely Most Expert Judgement Relevant Range by Analogy Sizing Database Functional Analysis SEER-AccuScope Delphi Analysis Expert Judgment Functional Analysis Multiple Comparisons Counts for Pre-existing Sizing Databases Using Multiple Methods Analogies 34

35 Why do a sizing study? By analyzing the size several ways, your estimate will be less influenced by outliers When multiple methods are used, often the results will cluster around particular values, thus giving credibility to your size estimate Using multiple methods and a selection criteria is usually more reliable and defendable than using a single sizing method 35

36 Document, document, document! Document all data sources! Document which experts were used, where the analogies came from, source of line/function counts, organizational size growth data Be certain to document all assumptions relating to: Language Language conversion factors Functionality included in the size estimate Rework Size growth Ownership of functionality (ex. will the security be handled by the purchased OS?) Document as you go if you don t you ll forget your rationale before you get it written down! 36

37 Sizing Study Example Task: Develop a size estimate for a particular program Resources: Development team Other development teams in the same company Software Process Improvement Group with expertise in the organization Relatively firm design in Rationale Rose Decent history of performance on similar programs in the past 37

38 Select Methods and Establish Groundrules Based on available time, data and other resources, the following methods have been selected: Expert Judgment Functional Measurement Delphi Analysis Analogy with existing programs Database Analysis AHP Comparison (SEER-AccuScope) Groundrules: All size estimates using various methods will be normalized to the intended development language, C++ SLOC estimates will be normalized to logical lines of code (standard SEER-SEM definition) Normalized Lines of C++ Method Least Likely Most Expert Judgement Functional Measurement Delphi Analysis Relevant Range by Analogy Database Analysis SEER-AccuScope Composite 38

39 Expert Judgment Lead developer was consulted and expressed opinion that this project will be about 50,000 lines of C++. Feels that this estimate is correct within plus or minus 10,000 lines of code. Normalized Lines of C++ Method Least Likely Most Expert Judgement Functional Measurement Delphi Analysis Relevant Range by Analogy Database Analysis SEER-AccuScope Composite 39

40 Functional Measurement Use cases were counted based on available design documentation. Metrics from the table below were used to convert use cases into C++ SLOC. The metrics from the table are felt to be accurate within -10/+20% The team is not confident that all use cases have been properly defined Use Case Size Number of Use Cases Lines of C++ per Use Case C++ SLOC Small Medium Large Total Normalized Lines of C++ Method Least Likely Most Expert Judgement Functional Measurement Delphi Analysis Relevant Range by Analogy Database Analysis SEER-AccuScope Composite 40

41 Delphi Study A Delphi study was conducted using a panel of six experts from within the development team, other development teams and the process improvement group. Five iterations were conducted, with results as shown below. Normalized Lines of C++ Method Least Likely Most Expert Judgement Functional Measurement Delphi Analysis Relevant Range by Analogy Database Analysis SEER-AccuScope Composite 41

42 Analogy Several analogous programs were selected based on similar platform, application and development environment. Comparison resulted in below range estimate. Normalized Lines of C++ Method Least Likely Most Expert Judgement Functional Measurement Delphi Analysis Relevant Range by Analogy Database Analysis SEER-AccuScope Composite 42

43 Database Analysis Thirty data points were selected from the sizing data repository based on similarity in application and development environment. Standard language conversion factors were used to normalize to C++. A normal distribution was assumed in selecting least, likely and most estimates. Normalized Lines of C++ Method Least Likely Most Expert Judgement Functional Measurement Delphi Analysis Relevant Range by Analogy Database Analysis SEER-AccuScope Composite 43

44 SEER-AccuScope Analysis Pairwise comparison was conducted using SEER-AccuScope. Six historical programs from the repository were selected which were similar in scope of functionality. Representatives from the development teams assessed comparisons. Results are as shown below. Normalized Lines of C++ Method Least Likely Most Expert Judgement Functional Measurement Delphi Analysis Relevant Range by Analogy Database Analysis SEER-AccuScope Composite 44

45 Analyze and Compare Methods 45 Expert Judgement Functional Measurement Delphi Analysis Relevant Range by Analogy Database Analysis SEER- AccuScope

46 Assess Least, Likely and Most Most Likely Least 46 Expert Judgement Functional Measurement Delphi Analysis Relevant Range by Analogy Database Analysis SEER- AccuScope

47 Sanity check Does your size estimate make sense? Sanity check the size of each module Look for outliers Pay attention (but don t totally rely on) developer s gut Sanity check the sum of the size of each module against the expected system size Is the sum of the pieces relatively in line with the expected size of the entire system? Sanity check the relative sizes of each module of the system under estimation Do the smallest pieces have the smallest size estimate? 47

48 How Big is My Software Project? 54,015 what? C++ logical SLOC Why not 30,000? Because there s no reason 54,015 plus or minus what? 24,721 to 78,950 to believe so 54,015 according to whom? Well, lots of people 54,015 with or without requirements creep? Without assuming built as specified 48

49 Thanks For Your Interest Contact Information: x x644 Copyright 2004, Galorath Incorporated. ALL RIGHTS RESERVED Reproduction, copy, or excerpt in whole or in part is strictly prohibited without written permission SEER is a trademark of Galorath Incorporated 100 N. Sepulveda Bl., Suite 1801, El Segundo, CA