STUDIES OF SOFTWARE PROCESS IMPROVEMENT. Summarized by Neal Coulter Updated April 15, 2003

Transcription

1 STUDIES OF SOFTWARE PROCESS IMPROVEMENT Summarized by Neal Coulter Updated April 15, 2003 Many researchers and practitioners have investigated the utility of software process improvement and other approaches to best practices. Some studies [Brown], [Jones], [McConnell] suggest general guidelines for process improvement without direct reference to CMM. Others [Bamberger] look for the essence of CMM within the larger structure. Reports of CMM-based assessment

2 and improvement are now common. A review of some studies helps to put the objectives of this research in focus. Studies of CMM-Based Software Process Assessment and Improvement While software process improvement models and related assessments are relatively recent investigations, several long-term studies validate vastly improved productivity, quality, and cost-saving realized from sustained CMMbased improvement programs. Companies reporting successes include Hughes Aircraft [Humphrey], Raytheon [Haley] and [Dion], Schlumberger

3 [Wohlwend], Motorola [Daskalantonakis], [Diaz] and Boeing [Yamamura], [Fowler], [Wigle]. Raytheon s Experiences The recently reported effort by Raytheon [Haley] provides a good case study for examining costs and benefits of process improvements. Raytheon began its process improvement efforts in 1987 and reached SEI Level-3 (from Level-1) in 1991; Raytheon continues its improvement efforts. Some impressive improvement results reported by Haley are:

4 In the two years prior to the initiative, rework costs averaged 41 percent of project costs. Two years later, rework costs were 20 percent of project costs and the trend continues downward. The major reduction in rework costs was associated with fixing source code problems found during integration, which dropped by 80 percent. Software productivity, as measured by delivered source instructions per person-month, has increased 190% since 1988, with most of that increase occurring by 1992.

5 In 1988, Raytheon s software projects actual costs exceeded budgeted costs by roughly 40 percent. By late 1991, the difference was in the plus or minus 3 percent range. Software quality, as measured by defect density, improved from an average of 17.2 software trouble reports per thousand of delivered source instructions to 4.0 reports; again, most of that decrease occurred by In a separate, earlier report on Raytheon s improvement program [Dion, 1993], return on investment of every dollar spent on process improvement was estimated at $7.70.

6 Haley identified two critical activities in Raytheon s improvement activities: 1. Raytheon established a strong and effective software process infrastructure for continuous improvement and to maintain enthusiasm over time. 2. Raytheon measured and analyzed process and project data to quantify benefits of software process improvement. To accomplish these objectives, Raytheon created an SEPG with four entities:

7 1. An executive steering committee to provide oversight and direction 2. Four working groups that each specialize in one of the major process-improvement disciplines 3. Ad hoc teams to develop the actual improvement processes 4. An SEPG manager to coordinate day-to-day activities The Raytheon SEPG s working groups are: 1. Policy and Procedures - Developed a three-tiered set of documents that define Raytheon s standard software process.

8 2. Training - Developed and continually upgrades a comprehensive training program 3. Tools and Methods - Responsible for selection, evaluation, and path finding of software development methods and tools 4. Process Database - Built a center for collecting and analyzing process data Haley summarized the reasons behind Raytheon s success as: Engineering leadership provided the vision and the commitment to success, linking process improvement with project performance.

9 The general manager supported the process. The process improvements clearly and continually demonstrated business benefits to projects. The SEPG carefully considered Raytheon corporate culture. The task managers and line engineers performed the vast majority of the work and hence felt ownership of the process and the products. This last reason--running the initiative from within the ranks of the software organization--was the most important.

10 Schlumberger s Experiences Schlumberger also pursued process improvement [Wohlwend] using CMM and reached Level-2. Some highlights reported there include: 94 percent of projects were completed on schedule in 1992, compared with 89 percent in 1991 and only 51 percent in 1990.

11 Another group saw average project completion improve from twice the estimated time to only a 1 percent slip against the original plan in a two and onehalf year period. At the start of this improvement work, the total line count for products was 400,000 NCSS (non-commented source statements) with an average defect rate of 0.22/KNCSS; in less than three years the corresponding counts were 700,000 NCSS and 0.13/KNCSS. In early 1989, 25 percent of total product defects were found by customers; by December 1991, the rate was 10 percent.

12 Lessons learned from Schlumberger include: Strong sponsorship by the chief executive officer is critical. A central, experienced team should participate in software improvement initiatives. In-house groups should be trained together to effect a cultural change. To accomplish these gains, Schlumberger devoted one to five fulltime staff members to process improvement in centers with 120 to 180 engineers; one to three full-time people are assigned in units of 50 to 120 engineers. While this commitment is large, the payoff is enormous.

13 Hughes Experiences Lessons learned from Hughes [Humphrey] include: Annual savings due to process improvement were estimated to be around $2 million annually. The path to improvement requires investment, risk, time, and the pain of cultural change. Delegation is not strong enough to overcome these roadblocks. Commitment is. Process improvement should be tied to the salary or promotion of senior management A focal point is essential...the SEI calls this focal point an SEPG. Hughes calls it an engineering council. Whatever the name, there must be a focal point.

14 Motorola s Experiences [Daskalantonakis] Lessons learned from Motorola [Daskalantonakis] include: Management buy-in is essential to a successful implementation of the progress assessment instrument and process. All SEI CMM sections for a given key process area, not just the Activities Performed, should be considered when using the scoring guidelines to determine a score, and when the list of action plans are created.

15 Ensure the entire progress assessment focus on identifying the organization s strengths and weaknesses (the findings) and...less on what a given activity s score should be. Common themes run through these four case studies. Process improvement is possible, it doesn t take forever, and it is cost effective. The lessons learned always are similar: Software processes can be improved if the issue is attacked aggressively.

16 Motorola s Experiences [Diaz] Motorola s Government Electronics Division (GED) was assessed at level 2 1n 1989, at level 3 in 1992, and level 4 in In 1995, GED s policies and procedures were rated level 5, without verifying project implementation of the level 5 KPAs. The relative improvements are shown in Table 1.

17 Table 1: Motorola s Improvements by Levels CMM Number Quality 1 Cycle Productivity 3 Level of Projects Time 2 (X factor) 1 3 n/a 1.0 n/a Defects per million assembly-equivalent lines of code.

18 2 Amount of calendar time for the baseline product to develop divided by the cycle time for the new product to develop. If the baseline product took six months to complete and the new product took two months to complete, then the X factor is 3. 3 Amount of work divided by the time to produce that work. Can be measured by lines of code. Lessons learned include: Biggest gains are made in peer inspections; Focus on improving new projects. It is extremely difficult to change projects, especially at lower maturity levels, once they have started.

19 Adopt a top-down focus before immersing yourself in CMM details; start by assessing the intent of each KPA so that you can determine how it fits into your environment; Emphasize productivity, quality, and cycle time. Avoid process for its own sake; Management commitment is needed from all levels; commitment from upper management won t be enough unless individual proj ect leaders and managers are determined to succeed; Practitioners and task leaders, not outside experts, should be used to define processes;

20 Managers must be convinced of process improvement s value; it s not free, but in the long run it more than pays for itself. The customer must be kept informed about the process, especially when process changes occur; Copying process documents from other organizations usually does not work well; the process must match your organization; Overcoming resistance to change is probably the most difficult rung to climb on the CMM ladder Advancing from level 2 to level 3 is most difficult because of the number of KPAs at level 3 and the impact process maturity plays in SPI.

21 The return on investment was estimated to be 677% from level 2 to level 5. This result was calculated as follows: Four full-time software engineers (from 350 software developers) 48 staff-months; Task leaders of the software improvement working group = 34 projects x 1 hour/week = 10.5 staff months; Prepahase kickoffs and post mortems: 34 projects x 1 hour prephase x 7 phases = 1.5 staff months; Software development planning; 34 projects x 5 days = 1 staff month; and

22 Defect prevention working group: 8 members x 1 hour per week = 2.5 staff months. From a quality perspective, the defect injection rate decreases by roughly half for every CMM level advanced. Therefore a level 2 organization has about eight times the defect rate of a level 5 organization. The cost of rework is at least eight times as great at level. Assuming each defect requires 16 hours analysis and rework and that the cost of rework is about $100 an hour, a typical 100,000 SLOC project can anticipate the following savings:

23 At level 5, 63 defects (based on an observed 126 defects per million earned assembly-equivalent lines of code expends $108,000 on rework. At level 2, 445 defects would expend $712,000 on rework.

24 A typical 100,000=SLOC project would span 18 months and employ approximately 20 engineers. If we allocate the 1.5% process improvement cost into each specific project, this would amount to 1.5% x 20 people x 18 months x 167 hours x $100/hour = $90,180 investment. The resulting return on investment would be ($712,000- $100,800)=$611,200 for a $90,180 investment, or a total return of 677% Motorola does not realize all these savings as profits, of course. The true cost-benefit occurs when projects finish earlier, allowing the acquisition and development of new business.

25 Boeing s Experiences In July 1996, the Defense and Space Group of Boeing Space Transportation Systems achieved a level 5 rating. Some of the results from [Fowler], [Wigle] and [Yamamura] are summarized here: Number of defects found during development increased to almost 100%; Customer satisfaction is always near 100%; Employee job satisfaction (scale of 1-10) increased from 5.7 to 8.3, with at 96% at least satisfied (as opposed to 74% earlier).

26 The key to Boeing s improvement is its SEPG. Some lessons learned there are: Charter the SEPG based on continuous process improvement and provide it real authority to make a difference. Do not let the SEPG falter after an assessment is completed; Members of the SEPG must be key leaders and domain experts from the projects in the organization and not outsiders. The process owners must make decisions about their processes to cause institutionalization; Provide sponsorship from all levels of management;

27 Be sure the SEPG understands its changing role as an organization matures; View an assessment as an opportunity for further process improvement; The SEPG s charter should include process definition, process change management, technology insertion, process evaluation, training, process improvement support, and process assessment; Empower the SEPG to make decisions within its charter with no further approval; Make SEPG members accountable.

28 General Dynamics Experiences The General Dynamics Decision Systems group employs about 1,500 engineers, with 360 of the directly involved in software development. By November 2001, all organizations within the group had achieved level 5 [Diaz and King]. Table 2 summarizes improvement trends for rework, phase containment, customer reported unique defects (CRUD), and productivity level.

29 Table 2: General Dynamics Project Performance Versus CMM Level CMM Percent Productivity Level Rework (X factor) Phase Containment Effectiveness CRUD Density per KSLOC % 25.5% X % 41.5% X 4 9.5% 62.3% X 5 6.8% 87.3% X

30 The cost for SPI was 2.5% of base. The calculated return on investment from avoided rework is: Level 2 to 3, 167%; level 3 to 4, 109%; level 4 to 5, 14%. These figures do not include the added benefit of applying existing resources to the pursuit of new business. Some lessons learned are: Management commitment is needed from all levels; commitment from upper management will not be enough unless individual project leaders are determined to succeed. Practitioners and task leaders, not outside process experts, should be used to define processes.

31 Managers need to be convinced of the value of process improvement. It is not free, but in the long run it certainly pays for itself. Overcoming resistance to change is probably the most difficult hurdle when climbing the CMM ladder. There are no silver bullets! Process change takes time, talent, and a commitment with which many organizations are uncomfortable.

32 Other Studies Surveys of CMM veteran organizations reveal the software process assessment has overall positive results for organizations that are successful in process improvement [Johnson], [Herbsleb]. The higher the CMM level, the more metrics the corresponding organizations collect--as implied by the quantitative dimensions of the higher levels. The KPAs most affecting schedule were Software Project Planning and Software Project Tracking and Oversight; the KPA most affecting quality was Peer Reviews [Johnson]. This survey covered 35 organizations that included all level 3, 4, and 5 organizations known then.

33 A survey of 167 individuals from 61 organizations [Herbsleb] revealed consistently positive return-oninvestment. While CMM based [software process improvement] is not a cheap fix, successful organizations showed high returns and enjoyed greater customer and employee satisfaction. These companies were less risk-averse than lower-level companies. No significant differences were found in CMM success as related to organizational size among the companies studied. Again, organizations with difficulties with the level a KPAs Software Project Planning and Software Project Tracking and Oversight often faltered at level 3.

34 However, everyone is not enthusiastic about software process assessment [Fayed], [Bach], [Bollinger] for numerous reasons. No known studies actually show negative return-on-investment figures for successful assessment, though. REFERENCES Bach, J. Enough About Process: What We Need are Heroes, IEEE Software, Vol:12, No:2, March 1995:96-98 Bamberger, J. Essence of the Capability Maturity Model, Computer, Vol:30, No:6, June 1997:

35 Brown, N., Industrial-Strength Management Strategies, Software,Vol:13, No:4, July 1996: Bollinger, T. and C. McGowan, A Critical Look at Software Capability Evaluations, IEEE Software, Vol:8, No:4 July 1991: M. Diax and J. King, How CMM Impacts Quality, Productivity, Rework, and the Bottom Line, Crosstalk, March M. Diaz and J. Sligo, How Software Process Improvement Helped Motorola, IEEE Software, Vol:14, No:5, September/October 1997: M. Daskalantonakis, Achieving Higher SEI Levels, IEEE Software, Vol:11, No:4, July 1994:17-24.

36 R. Dion, Process Improvement and the Corporate Balance Sheet, IEEE Software, Vol:10, No:4, July 1993: Fayed, M and M. Laitinen, Process Assessment Considered Wasteful, Communications of the ACM, Vol:40, No:11, November 1997: K. Fowler, SEI CMM Level 5: A Practitioner s Perspective, Crosstalk, September T. Haley, Software Process Improvement at Raytheon, IEEE Software, Vol:13, No:6, November 1996:33-41.

37 J. Herbsleb, D. Zubrow, D. Goldenson, W. Hayes and M. Paulk, Software Quality and the Capability Maturity Model, Communications of the ACM, Vol:40, No:6, Jun 1997: W. Humphrey, T. Snyder, and R. Willis, Software Process Improvement at Hughes Aircraft, IEEE Software, Vol:8, No:4, July 1991: D. Johnson and J. Brodman, Realities and Rewards of Software Process Improvement, IEEE Software, Vol:13, No:6, November 1996: C. Jones. Our Worst Current Development Practices, Software,Vol:13, No:4, March 1996:

38 S. McConnell, Software s Ten Essentials, Software, Vol:14, No:2, March/April 1997: H. Wohlwend and S. Rosenbaum, Schlumberger s Software Improvement Program, IEEE Transactions on Software Engineering, Vol:20, No:11, November 1994: G. Wigle and G. Yamamura, Practices of an SEI CMM Level 5 SEPG, Crosstalk, November G. Yamamura and G. Wigle, SEI CMM Level 5: For the Right Reasons, Crosstalk, August 1997.