Experience Report on COCOMO and the Costar Tool from Nortel's Toronto Laboratory

Transcription

1 Experience Report on COCOMO and the Costar Tool from Nortel's Toronto Laboratory Danny Ho, Northern Telecom Canada Limited Abstract Northern Telecom Canada Limited (Nortel*) is the largest telecommunications company in Canada. Nortel manufactures telecommunication systems and related equipment. Nortel's Toronto Laboratory is responsible for the development of multimedia communication systems including messaging applications and call center applications. Software estimation in the Toronto Laboratory is done in a traditional manner and the project office keeps track of project actuals such as time spent per month by individuals on development. This paper captures the experience of calibrating and utilizing the Constructive Cost Model (COCOMO) and the Costar* tool. Historical data available from the project office was first used for calibration of the Costar tool for realtime telecommunications software development in Nortel's Toronto Laboratory. The exercise included a small call center application and three releases of a large messaging application, covering different languages and platforms. Upon successfbl calibration, new development began to use COCOMO to validate bottomup estimates, Project estimation on the limited adoption of objectoriented technology was also discussed at the beginning of the project and six months later in that project's development lifecycle. 'The report concludes with the key challenges, lessons learned, and future directions in utilizing estimation models and tools. 1 ntroduction The software development lifecycle in Nortel's Toronto Laboratory consists of analysis, design, coding, testing, and maintenance. Gates (or checkpoints) are defined to verify and qualify stages of development. Gate 0: Decision made to launch a project. Gate 1 : Development team completes planning, specification, and highlevel design. Gate 1 A: Development team completes lowlevel design, coding, and unit testing. Gate 2: Test team completes integration and system tests. The product is ready for first Nortel is a trademark of Northern Telecom Costar is a trademark of Softstar Systems

2 customer ship. Gate 2A: Field trials are complete. Product is ready for shipment to enduser sites. Gate 3: Product is ready for sustaining process and volume shipment. Gate 4: Product met its objectives following standard production and service. Software estimation for the project is done after Gate 1. The following diagram shows the estimation logistics. assumptions w component 1 dependencies 1 objectives Estimation... development Master Project consensus approval development plan calendar Figure 1 Estimation Logistics Although the process for project estimation is welldefined, objectives have been set for more automation in performing whatif analysis on the overall estimate, and for validating the bottomup estimates of various components in a project. t follows that investigation using estimation models and tools would be necessary. COCOMO [] was first introduced to the Toronto Laboratory in one of our biweekly Laboratory Technical Seminars. The Costar tool [2] by Softstar Systems was demonstrated to laboratory personnel on several occasions. Following formal acquisition of the Costar tool, the project office was approached to obtain actuals captured in past projects. The actuals were used to calibrate the model and tool after interviewing and consulting with managers and key people of the past projects. The calibration process for a project is illustrated in Figure 2. n the following sections, details are presented in calibrating the model and tool from past projects, and utilizing the model and tool for validation of the bottomup estimate of a new project. At the end of the exercise, we will have compared project actuals to the COCOMO estimate, calibrated COCOMO effort multipliers for different projects, and adopted an estimation model for validating the bottomup estimates of future projects. An additional objective of the exercise is to provide feedback to executives and senior managers on challenges relating to project development.

3 select project obtain actuals from proiect office + determine effort, duration and peak staffing discuss project specifics with project office measwe, interview project managers and key developers summarize v document result and experiment comparison to with Costar b of project executives on estimate actuals with or senior ' lestimate pagers Figure 2 COCOMOCostar Calibration Process 2 Calibrating COCOMOCostar from past releases of a large project 2.1 Base release Project description The project involved enhancement of a realtime messaging application. The application consisted of approximately 596 KLOC (thousand lines of code) of an existing system, and 13 1 KLOC of new development. Approximately 5% code and design changes were made to the old code, and 25% integration effort was required. The primary language was Pascal. The development platform was UNX*basedand there were six proprietarytarget platforms.theproject office tracked theactuals shortly after Gate 1 (2Q92); and continued tracking even after Gate 3 (2994) was passed. Actuals were tracked in terms of persondays per month spent on the project, and staffing within each month. Persondays were then divided by the number of workdays in each month for conversion to staffmonths spent on the project. Managers and key people of the project were interviewed and consulted to establish ratings for the COCOMO effort multipliers Estimate versus actuals The following table summarizes the key input parameters to the Costar tool for the base release. Table 1: Key input parameters of the base release Parameter nput Database COCOMO 87 UNX is a trademark of Novel1 nc

4 Table 1: Key input parameters of the base release Parameter Development mode Semidetached nput Old: 596, 5% design, 5% code, 25% integration New: 13 1 The following table summarizes the effort multipliers and their ratings. Table 2: Effort multipliers for the base release Rating Effort Multipliers High LLow ACAP, PCAP, VEXP, LEXP, CPLX, TME, STOR, TOOL, RUSE DATA, VMVH, VMVT, SCED All other effort multipliers had nominal ratings. No userdefined effort multipliers were used. Due to past experience and maturity in development, good ratings were given to many effort multipliers. The following table shows the development summary and compares the estimate with actuals. Table 3: Development summary of the base release Output parameters Duration (months) Effort (staffmonths) Estimate without effort multipliers Estimate with effort multipliers Actuals Peak staffing Observations As can be seen fiom the above table, the estimate with effort multipliers is improved over the one without effort multipliers (or all effort multipliers with nominal ratings). The duration was shortened by approximately 26%, and the effort decreased by 27%. When comparing the estimate with effort multipliers to the project actuals, the duration was within 10% of the actual, and peak staffing was very close. However, there was a 1,arge discrepancy in the effort. The following accounted for this discrepancy. (1) Actuals did not record overtime effort. (2) Actuals did not include effort spent by the project office, configuration management, quality assurance, documentation, and the management staff. (3) Actuals were tracked after Gate 1, thus missing the effort spent in planning, specification,

5 and highlevel design. By adding the effort estimated by COCOMO for (2) and (3) to the actuals, the effort should be approximately 580 SM (staffmonths). Assuming 10% overtime effort, the total effort should be around 638 SM, which is 27% lower than the overall COCOMO estimate. Although there is no fair comparison between the effort of the estimate and actual, the discrepancy can be explained. 2.2 First update to the base release Project description The first update to the application was developed in t was a minirelease for fixing bugs in the base release. Based on the COCOMO adaptation estimating output, we used 197 KLOC as the size of the base release. Approximately 10% design changes and 25% code changes were made, and 30% integration effort was required. The size of new development was 11.4 KLOC. The development language and platforms remained. Actuals were tracked shortly after Gate 0 (4494) and continued tracking after Gate 2 (3495) Estimate versus actuals The following table summarizes the key input parameters to the Costar tool for the first update. Table 4: Key input parameters of the first update Parameter nput Database COCOMO 87 Development mode KDS Semidetached Old: 197, 10% design, 25% code, 30% integration New: The following table summarizes the effort multipliers and their ratings. Table 5: Effort multipliers for the first update High Low Rating Effort Multipliers ACAP, AEXP, PCAP, VEXP, LEXP, CPLX, STOR, TOOL, RUSE DATA, VMVH, VMVT, SCED All other effort multipliers had nominal ratings. No userdefined effort multipliers were used.

6 The following table shows the development summary and compares the estimate with actuals. Table 6: Development summary of the first update Output parameters Duration (months) Effort (staffmonths) Peak staffing Observations Estimate with effort multipliers Actuals As can be seen from the above table, the duration was close, and the effort was within 20% of the actual. Project tracking was started earlier in the development lifecycle, and the results of comparison between the estimate and actuals showed improvement over the base release. However, peak staffing was not comparable. By interviewing managers and key people of the project, we learned that the staff might not be working solely on the current release. 2.3 Second update to the base release Project description The second update to the application was developed in conjunction with the first update. These releases were originally treated as one. t was later decided to supply the first update for fixing bugs in the base release and incorporating additional key features. Based on the COCOMO adaptation estimating output, we used KLOC as the size of the first update. Approximately 10% design changes and 20% code changes were made, and 30% integration effort was required. The size of new development was 54.3 KLOC. The development language and platforms remained. Actuals were tracked shortly after Gate 0 (2494) and the remaining estimate was validated around Gate 1 A ( 1 (296) Estimate versus actuals The following table summarizes the key input parameters to the Costar tool for the second update. Table 7: Key input parameters of the second update Database Parameter Development mode KDS COCOMO 87 nput Semidetached Old: 51.7, 10% design, 20% code, 30% integration New: 54.3

7 The following table summarizes the effort multipliers and their ratings Table 8: Effort multipliers for the second update Rating 1 Effort Multipliers High 1 VEXP, CPLX, STOR, TOOL, RUSE Low DATA, VMVH, VMVT All other effort multipliers had nominal ratings. No userdefined effort multipliers were used. Lower personnel ratings were given due to transfer of experienced staff to other projects and addition of new staff to the current release. The following table shows the development summary and compares the estimate with actuals. Table 9: Development summary of the second update Output parameters Duration (months) Effort (staffmonths) Peak staffing Observations Estimate with effort multipliers Actuals As can be seen from the above table, the duration was within approximately one month, effort was within 17%, and peak staffing was within 24% of the actuals and the remaining bottomup estimate. The result of comparison between the estimate and actual showed improvement over the previous release. This was evidence of better project tracking and calibration of COCOMO by establishing effort multipliers for releases of the same project. 3 Calibrating COCOMOCostar from a small project 3.1 Project description The project involved enhancement of a realtime call center application. The application consisted of i3pproximately 84.4 KLOC of an existing system, and 1.25 KLOC of new development. Approximately 2% design changes and 5% code changes were made to the old code. and 40% integration effort was required. The primary language was C. The development platform was UNXbased. Actuals were tracked from 1 Q94 to Estimate versus actuals

8 The following table summarizes the key input parameters to the Costar tool for the small project. Table 10: Key input parameters of the small project Parameter / nput bat abase COCOMO 87 1 Development mode ( Semidetached 1 Old: 84.4, 2% design, 5% code, 40% integration New: 1.25 The following table summarizes the effort multipliers and their ratings. Table 11: Effort multipliers for the small project Low VMVH, VMVT, MODP All. other effort multipliers had nominal ratings. No userdefined effort multipliers were used. The following table shows the development summary and compares the estimate with actuals. Table 12: Development summary of the small project Output parameters Estimate with effort multipliers Actuals Furation (months) 11.1 Effort (staffmonths) /, Observations As can be seen from the above table, the duration was within approximately one month, effort was within 8%, and peak staffing was within 25% of the actuals. Calibration of COCOMO on the small project was successful. 4 Estimating a new project with limited objectoriented development 4.1 nitial estimate Project description The project involves new development of a realtime call center application of approximately 98

9 ~ KLOC. The primary language is C++. The development platform is Windows NT*. The development team has adopted the Booch Methodology [3] in a limited fashion to perform objectoriented analysis and design of the system. The Rational Rose*/C++ tool [4] has been acquired to implement the Booch notations. At the time the initial estimate was validated with COCOMO, the project had passed Gate 1. Project estimation and scheduling then followed. The bottomup estimate was submitted using the following key parameters for different iterations. (1) Number of classes (2) LOC (3) Effort for development in persondays Managers of the project were interviewed to establish the ratings for the COCOMO effort multipliers Estimate versus bottomup The following table summarizes the key input parameters to the Costar tool for the project. Table 13: Key input parameters of the new project Parameter nput Database COCOMO 87 / Development mode 1 Semidetached KDS 98 Using the ratings for effort multipliers of the past projects as the baseline, ratings for personnel parameters were lowered to reflect development using an objectoriented approach for the first time. Ratings for product parameters are the same, except that the reliability requirement of call center applications is higher than messag'ing applications, and the size of the database is large. Ratings for computer parameters are improved due to deployment of Pentium* PCs, and gradual maturity in the development environment. Ratings for project parameters are the same as the baseline, except that more tools are utilized in development. The final COCOMO output may be finetuned manually to allocate more effort in analysis and design, and less incoding and testing. The following table summarizes the effort multipliers and their ratings. Table 14: Effort multipliers for the new project Rating / Effort multipliers Very high High TOOL RELY, DATA, CPLX, RUSE Windows NT is a trademark of Microsoft Corporation Rational Rose is a trademark of Rational Software Corporation Pentium is a trademark of ntel Corporation

10 Table 14: Effort multipliers for the new project Cow Rating Effort multipliers AEXP, VEXP, LEXP, VMVH, VMVT / Very low / TURN All other effort multipliers have nominal ratings. There are no userdefined effort multipliers. The following table shows the development summary and comparison with the bottomup estimate. Table 15: Development summary of the new project Output parameters Estimate without effort Estimate Bottomup estimate Duration (months) Peak staffing Observations As can be seen from the above table, the estimate with effort multipliers is higher than the one without effort multipliers. The duration is lengthened by approximately 5%, and the effort increases by 16%. Lower ratings for the personnel parameters as a result of firsttime objectoriented development are partially offset by higher ratings for the computer parameters. When comparing the estimate with effort multipliers to the bottomup estimate, the duration lies within 7% of the bottomup, and peak staffing is very close. Again, there is discrepancy in the effort. By adding the effort estimated by COCOMO for the project office, configuration management, quality assurance, documentation, and the management staff, the total effort of the bottomup estimate becomes 61 0 SM, which agrees with the estimate with effort multipliers. Note that as a rule of thumb for objectoriented software development [5], the effort for analysis and design should be increased by 20%, coding decreased by 20%, and integration decreased by 3094 from the traditional estimate. The actuals for this project will serve to check whether this rule also applies to Nortel's Toronto Laboratory. 4.2 Revisiting the estimate after six months Estimate versus actuals and the remaining bottomup estimate The estimate was revisited six months later. Management requested that duration of the project be shortened, and staffing be increased. The COCOMO estimate was modified by giving a low rating to the SCED effort multiplier. At the same time, the project office was able to supply project actuals tracked shortly after Gate 0. The bottomup estimate was extended to include indirect effort. Thus,

11 the COCOMO estimate is directly comparable with the project actuals and the remaining bottomup estimate. The following table shows the development summary and comparison with the project actuals and thle remaining bottomup estimate. Table 16: Development summary of the new project after six months Output parameters Estimate with effort multipliers Actuals and remaining bottomup estimate Duration (months) Effort (staffmonths) Feak staffing Observations As can be seen from the above table, the duration is very close. The effort is within 9% and peak staffing is within 12% of the actuals and the remaining bottomup estimate. t is concluded that periodic revisit of the estimate would bring convergence on the actuals and the remaining bottomup estimate. 5 Lessons Learned Th~experience in calibrating and utilizing COCOMO and the Costar tool for past projects and new projects in Nortel's Toronto Laboratory has been described in this report. We have used COCOMO to validate projects of various sizes, languages (e.g., Pascal, C, C*), and platforms (e.g., UNX, Windows NT):COCOMO effort multipliers have been established for various projects. n the exercises for both past projects and new projects, the COCOMO estimates agreed with the actuals and the bottomup estimates. All discrepancies can be explained. n order to minimize discrepancies in future estimates, the developmentbased estimation logistics have been extended to cover indirect effort. n addition, project actuals are being tracked after Gate 0, instead of after Gate 1, to include effort spent in planning, specification, and highlevel design. Overtime should also be reported promptly. For example, a developer who works overtime may choose to save the time for use after the project is over. f the time saved is not reported as overtime, tracking of the actuals will become inaccurate. COCOMO and the Costar tool have been adopted for labwide deployment. We would like to extend the scope to the whole company in the future. The use of COCOMO and the Costar tool will be complementary to the bottomup estimate. t should be noted that they help validation of the bohtomup estimate, but do not replace the process of detailed scheduling. Since no estimation model is perfect [6], the use of more than one is highly desirable. This paper will form the basis for repeating the calibration and estimation exercises on other estimation models and tools in Nortel's Toronto Laboratory.

12 References [ ] Barry Boehrn, Sof~are Engineering Economics, PrenticeHall, Englewood Cliffs. NJ, [2] Softstar Systems, Costar User's Manual (Version 4), Amherst, NH, [3] Grady Booch, ObjectOriented Analysis and Design with Applications, Second Edition, BenjaminKummings, Redwood City, CA, [4] Rational Software Corporation, Using Rational Rose 3.0, Santa Clara, CA, [S] Louis Ham, Mohamed Fayad, Managing ObjectOriented SoJiware Developmrnt Projects, Object Technologies nc., [6] Nikki PanlilioYap, Danny Ho, Lessons Learned in Deploying Sofh~are Estimation Technology and Tools, Nineteenth Annual Software Engineering Workshop, NASA/ Goddard Space Flight Center, Maryland, Acknowledgements The author would like to thank individuals from the product development groups who provided project data for this report, Ed Kole who encouraged the writing of this report, Clifford Yeung who provided assistance in the formatting system, and Taylor Ledden who provided editorial improvements. About the author Danny Ho works in the Software Development Environment Group in Northern Telecom Canada Limited. His areas of special interest are software estimation, objectoriented software development, and complexity analysis. Prior to joining Northern Telecom, Danny worked as a team leader in infrared wireless development in the BM Microelectronics Toronto Laboratory, and in the Tools and Technology Group in the BM Software Solutions Toronto Laboratory. He also had experience in developing wireline and radio frequency communication systems, while working as a software engineer in the Communications Division of Motorola Canada Limited. Danny received his Honors Bachelor of Science in Computer Science and Electrical Engineering, and Master of Science in Computer Science from the University of Western Ontario. He is currently a member of the Professional Engineers Ontario (PEO). Danny can'be reached at Northern Telecom Canada Limited, 522 University Avenue, Toronto, Ontario M5G 1 W7, Canada. His address is hod@bnr.ca.