Software Metrics "State of the Art"

Transcription

1 Summary Software Metrics "State of the Art" Peter Kulik, KLCI This paper reports the results of KLCI s worldwide survey of software metrics practices, conducted in the fourth quarter of Software metrics considered are both traditional "quality" metrics as well as process and project metrics. Based on survey responses, the most commonly used metrics (used by at least of respondents) are summarized in Table 1: 1 Schedule Metrics (55%) Lines of Code (46%) Schedule, quality, and cost tradeoffs (38%) Requirements Metrics (37%) Test Coverage % (36%) Overall Project Risk % (36%) Figure 1 - Most Commonly Used Metrics Of the 256 organizations that participated in our research, 37% rated software metrics as an important or integral part of their software projects, as shown in Figure 2; these organizations are considered to represent " s" for software metrics. practices organizations differ from other software organizations in several key areas: Keep more metrics Select metrics that provide project insight and are consistent across multiple projects Are more tool-oriented Undertake more process improvement initiatives Have senior and QA organizations who are more involved in software metrics Are more likely to be located in India, Australia / New Zealand, or the US. These findings are discussed in more detail in the following sections % 1. How important are measurements/metrics to your software development activities? 11% 33% 1 - Minimal One of Several Factors 16% Figure 2 Importance of Software Metrics 21% Integral 1 For all tables, numbers in parentheses indicate the percent of respondents choosing the item listed. Copyright 2000, KLCI

2 Demographics 256 software development organizations worldwide participated in this study. As shown in Figure 3, respondents represented a variety of organizations - from less than 15 to more than 100 developers, developing a wide variety of software, both US-Based and non-us-based. Further, 8 of respondents have some level of management responsibility. The study was conducted using electronically distributed questionnaires, sent to individuals registered at KLCI s website (), and posted in relevant Usenet newsgroups. Respondents were self-selecting; survey results include data from large companies such as Raytheon, Siemens, and Intel; government agencies such as NASA; and a variety of medium and smaller companies such as Collective Wisdom and Computer Generation. Which of the following best describes your position? Executive 1 Senior Manager 14% Manager Developer Project Leader 3 How many projects does your organization typically complete in one year? More than to to 1 4% 2 to 5 How many software developers are in your organization? 1 to 5 9% 6 to 15 1 More than to to 50 11% 16 to 30 What type of software does your organization develop? Object- Oriented Other 6% Mainframe 9% Client- Server 11 to 20 16% 6 to 10 21% Where is your primary development site located? Eastern Europe Other Africa South Asia America Australia / New Zealand India US/Canada Internet PC 17% Western Europe Figure 3 - Survey Demographics Copyright 2000, KLCI Page 2

3 Profile of Metrics s For the purposes of this study, best-practice organizations are defined as those who reported that software metrics are an important or integral part of their software projects, as shown in Figure 2 (page 1). The metrics kept by best-practice organizations and others are shown in Figure Which of the following quality measurements / metrics do you currently use? % 51% 51% 46% 54% 3 46% 45% 47% 28% 42% 33% 26% 18% 26% 9% 14% 14% 6% Schedule metrics Lines of code Requirements metrics Test coverage % Overall project risk % Fault density Function points Fault arrival and close rates Fault severity distribution Schedule contingency Teaming effectiveness Mean time to failure Cyclomatic complexity Figure 4 - Metrics s As summarized in Figure 5, the metrics kept by best practice organizations span both traditional QA metrics and process metrics such as requirements and project risk. practice organizations also typically keep more metrics than others an average of 7 metrics per organization, compared to an average of 4.5 metrics for all others. " " Organizations Other Organizations Schedule metrics (69%) Lines of code (51%) Requirements metrics (54%) Schedule metrics (51%) Fault density (47%) Test coverage % () Lines of Code (46%) Overall project risk % () Test coverage (46%) Requirements Metrics (3) Overall project risk % (45%) Function points (42%) Fault arrival and close rates (33%) Figure 5 - Comparison of Top Metrics for " " and "Other" Organizations Copyright 2000, KLCI Page 3

4 Criteria for Choosing Metrics When asked why the specific metrics identified were chosen, participants responded as shown in Figure Why did you choose the measurements / metrics selected in question 2? % 69% 45% 44% 56% 4 49% 49% 4 39% 28% 33% 18% Understood by management Provide insight into project activities Easy to implement Understood by software developers Historical reasons Consistently across multiple projects Dictated by mgmt or "corporate" processes 11% Predictive of customer experience Figure 6 Metrics Criteria practice organizations choose metrics using criteria different from those used by other organizations. While best practice organizations prioritize insight into project activities and consistent application across multiple projects, other organizations top criteria are ease of implementation and understanding by management. As shown in Figure 7, senior in best practice organizations are more involved in choosing metrics, while other organizations tend to leave this decision to QA organizations and individual software teams. 8. In your organization, who decides what software metrics are measured? % 27% 36% 22% 23% 18% 14% 14% 13% QA organization Software development Senior technical Individual project Executive 13% 8% Corporate staff organization Individual software developers 8% Other Figure 7 Metrics Decision-Makers Copyright 2000, KLCI Page 4

5 Metrics Tools As shown in Figure 8, best practice organizations are more likely to use tools to assist with software metrics than are other organizations. practice organizations also use more tools an average of nearly 2 tools each, compared to 1.3 tools for each other organization What tools does your organization use regularly to capture and analyze software metrics? % 63% MS-Excel from Microsoft 22% In-House Proprietary 19% RationalSuite - Rational Software 9% ClearQuest - Rational Software 6% 5% 6% 4% 2% 4% 1% 3% Function Point Workbench - Charismatek Public Domain tool Figure 8 Metrics Tool Usage McCabe toolset - McCabe & Assoc. QSM-SLIM - QSM 17% 1 Other Overall satisfaction with metrics tools was low, averaging just 2.6 (i.e. somewhat dissatisfied) on the scale of one to five. Users of a greater number of tools reported generally higher satisfaction, suggesting that a portfolio of tools is better suited to their needs than any single tool. Responses are shown in Figure 9: How satisfied are you with the software metrics tools your organization is using? ' 28% 8% 8% 27% 3% 13% 4% 1% Extremely Dis- Neutral Extremely Satisfied Satisfied Figure 9 Metrics Tool User Satisfaction While the most common tools are MS-Excel-based and in-house proprietary tools, users of packaged metrics tools generally reported higher satisfaction. Function Point Workbench scored the highest level of user satisfaction, at 3.9 (i.e. somewhat satisfied) on a scale of one to five, while MS-Project had the lowest level of user satisfaction, scoring 2.0 (i.e. somewhat dissatisfied). Users of the most common commercial packaged tools reported levels of satisfaction of approximately Neutral, which is the same level of satisfaction as reported by users of public domain tools. Copyright 2000, KLCI Page 5

6 Year-Over-Year Comparison Little change was seen year-over-year in the importance of metrics in software development, as shown in Figure 10. Importance of Metrics in Software Projects Year-over-Year Comparison % of Participants Selecting 3 1 5% 1 - Minimal One of Several Factors Integral Importance Figure 10 Year-over-Year Comparison The top process improvement initiatives undertaken by participants in 1999 were reported to include: Improve project management (49%) Better manage software processes (4) Increase use of metrics (31%) These initiatives have driven some changes in the mix of metrics kept by software organizations, as shown in the following comparison for best practice organizations: Metric Organizations Change in Usage Schedule Metrics +5% Lines of Code -1 Requirements Metrics +4% Test coverage % +5% Overall project risk % +9% Fault density +11% Function Points +3% Fault Arrival and Close Rates -3% Table 11 - Changes in Individual Metrics The general trend among best practice organizations appears to be towards capturing an increasing number of metrics, which is consistent with the improvement initiatives identified above. Copyright 2000, KLCI Page 6

7 Conclusion Software metrics can be used as a key part of software project management to improve the performance of software development groups. practice organizations are those that use metrics as an important or integral part of their software projects. Whether improving a current metrics program or establishing a new one, all organizations can learn from the practices of these best practice organizations: Ensure that metrics programs support your organization s business objectives Choose metrics that provide insight into project activities and can be used across multiple projects. Use metrics in project management decision-making Involve senior management Link metrics and process improvement initiatives About the Author Peter Kulik works with software organizations and project teams to implement software process improvement initiatives, especially in the areas of Requirements Management, Software Metrics, and Software Risk Management. As Managing Director of KLCI (), he has published widely on various software project management topics. Some recent clients have included ABP, Compaq, Guide Datakonsult AB, Halifax plc, NCR, Royal Bank of Canada, and Siemens. Resources Recent articles: Kulik, Peter J., A Practical Approach to Software Metrics," IEEE IT Professional, Jan/Feb Chikofsky, Elliot, and Rubin, Howard, "Using Metrics to Justify Investment in IT", IEEE IT Professional, Vol. 1 No. 2, March/April Dekkers, Carol A., "Tame your Process with Metrics", Enterprise Development, Vol. 1 No. 7, June Kulik, Peter J., Software Metrics s, Software Q/A Magazine, Vol.5 No.2, April/May Payne, Jeffery E., "Quality meets the CEO", Software Testing & Quality Engineering, Vol. 1 No. 3, May/June Weigers, Karl, "A Software Metrics Primer", Software Development, Vol. 7 No. 7, July Classic texts: Grady, Robert B., and Caswell, Deborah L., Software Metrics: Establishing a Company-Wide Program, Prentice-Hall, Inc., Putnam, Lawrence H., and Myers, Ware, Measures for Excellence, Prentice Hall - Yourdon Press, 1992 Copyright 2000, KLCI Page 7