Continuous Process Improvement - Why Wait Till Level 5?

Transcription

1 Continuous Process Improvement - Why Wait Till Level 5? Girish Seshagiri Advanced Information Services, Inc. Peoria, IL USA Abstract Continuous improvement is generally considered to be a paradigm shift that occurs when organizations move from the Managed to the Optimizing Process (SEI CM Level 4 to 5). This paper supports the view that continuous improvement needs to start early to enable and guide the gradual evolution and maturation of process definition across Key Process Areas and maturity levels. It is a well known principle of process change that Major changes start at the top, Besides changes starting at the top, as was the case in our organization, we needed to address two major organizational behavior tendencies in our evolution f%om SEI C34M Level to Level 2:. Viewing process definition as an all-or-nothing proposition instead of a gradual evolution, 2. Waiting for few large changes instead of implementing many small incremental changes. Data from our company s efforts is presented to support the following conclusions: * Continuous improvement as a way of life need not wait until organizations acquire ability to measure, verijfi! and improve processes at higher maturity levels. * A simple and effective mechanism can be implemented to enable and guide the gradual evolution and maturation of process dejinition even in low maturity organizations. * Tendency to view process definition as an all-or-nothing proposition can be overcome by empowering practitioners to implement changes and by providing rapidfeedback. * Of$cial, perceived, and actual processes begin to converge when practitioners suggestions are implemented in an institutionalized way. * Starting to define processes is itself the maj.or improvement activity in Level I organizations. Introduction Continuous improvement is generally considered to be a paradigm shift that occurs when organizations move from the Managed to the Optimizing process (SE CMM Level 4 to Level 5). The AIS experience supports the view that continuous improvement needs to start early to enable and guide the gradual evolution and maturation of process definition across Key Process Areas &PA) and maturity levels. In this paper, we present data from AIS process improvement efforts as evidence of what works. During , AIS engineers submitted 67 individual proposals for process improvement. Of these, 0 were implemented, were in progress, 52 were returned, and 3 were duplicates. Following are the details regarding our approach, the results and our conclusions: Background AIS is a custom contract software development company. During , AIS staff grew from to 22. While sales revenue gradually increased, company profitability remained low and somewhat unpredictable. Individual project results varied significantly. Our success depended on individual heroic efforts, typical of a CMM Level organization $ IEEE 68

2 Process Improvement - The Initial Approach In January, 992 we started an improvement initiative based on the SE CMM as the process model [. Additionally, we used Watts Humphrey s Managing the Software Process (2 book as our guide. Our approach was to: * Conduct self assessments with a focus on action. * Utilize skill and experience of many people, part-time, to implement findings and improvements. * Maintain organization awareness of improvement efforts through quarterly status reviews. Self Assessment As a small company, we could not afford formal assessments. Instead, we decided to use SE questionnaire TR-23 [3] followed by project team interviews. In January, 992 we conducted our first self assessment utilizing this approach. The TR-23 questionnaire answers (% of staff responding yes ) are given in Table : Table. Responses to TR-23 - Jan 92 No. Question Jan Does the Software Quality Assurance (SQA) function have a reporting channel 5 separate from development management?..6 Is there a software configuration control function for each project? Is a formal procedure used in the management review of each software development 6 prior to making contract commitments? 2..4 Is a formal procedure used to assure periodic management. review of the status of 26 each software project? 2..7 For each project, are independent audits conducted for each step of the software 0 develonment nrocess? 2..4 Is a formal procedure used to make estimates of software size? 2..5 Is a formal procedure used to produce software development schedules? Are formal procedures applied to estimating software development cost? 2.2. Are software stafiing profiles maintained of actual staffing versus planned staffing? Are software trouble reports resulting from testing tracked to closure? Does senior management have a mechanism for regular review of status of software 6 Not surprisingly, we concluded that we were identified the major findings and prioritized at CMM Level. Project team interviews them as shown in Table 2: Table 2. Prioritized findings opment environment 682

3 We then mapped the findings to Humphrey s suggestion for the way out of the ad hoc level [2] and SE CMM Level 2 Key Process [l] as shown in Table 3 : Table 3. Mapping the findings Way out of ad hoc CMM level 2 KPAs Project Management System Project Planning Project Tracking Software Configuration Management Commitment review { ) { 3 } Quarterly status review {S} Phase review { 6) Project plan process { 2) Project tracking process { 7) LAN documentation { 5) Focus On Action We followed up the self assessment and findings with specific actions.. Named our initiative Continuous Process Improvement (CPI). Our CPI concept was a continuous loop of assessing current practices, planning process improvements, and implementing process improvements. 2. Created a CPI long range plan. The plan set a target date of June 993 to achieve CMM Level 2. The plan showed specific milestone dates for process definition, routing, acceptance and implementation. 3. All employees were given a CPI manual as the repository for process definitions, policies, and standards. 4. All employees were given Watts Humphrey s Managing the Software Process book [. 5. Created a reference library including IEEE Software Engineering Standards [4], and relevant SE Technical Reports. Findings Implementation With Part Time Staff Effort Our implementation approach was to utilize the skill and experience of many people part time. We created a Software Process Engineering Group (SEPG). Project staff were assigned to the SEPG on a part time basis based on their availability and project schedule constraints. The SEPG was assigned the initial task of creating high level process definitions for commitment, project planning, and management review. During staff hours were recorded for SEPG activities, roughly equivalent to about 4% of AIS total staff hours. Quarterly Status Reviews We followed Humphrey s suggested agenda for quarterly status reviews. As the Senior Executive, I opened the meeting with a review of AIS financials and general comments pertaining to the importance of the CPI initiative and AIS long range goals. Projects presented project status, issues and progress against plans. SEPG presented CPI long range plan status. Computing Support presented status of hardware/software plans and project related technical issues. We held our second quarterly review on June 30, 992. The SEPG report on CPI indicated the following:. Process definitions for pre-commitment review, and project phase reviews were completed, routed and accepted. 2. Standards for work breakdown structure, and software quality assurance plans were defined, routed and accepted. 3. Process definitions for project planning, project tracking, and AIS LAN documentation and configuration management were in progress and behind schedule. It was apparent that while we were making progress, we were also falling behind in our schedule to reach level 2 by June 993. The results of self assessment conducted in July, 992 showed we had made very little progress in our improvement initiative. The responses to TR-23 questiomraire in July 992 compared to those from January 992 are shown in Table 4: 683

4 Table 4. Responses to TR-23 - Jan 92 & Jul 92 (Note: % yes answers actually declined in some instances. The good news was we now understood the questions better!) The slow progress seemed to result from two related organization tendencies:. Viewing process definition as an all or nothing proposition. Project Plan Process, for example, had gone through several drafls and was far from being accepted by the project teams. 2. Waiting for few large changes instead of implementing many small incremental changes. I realized that we had to overcome these organization tendencies to successfully implement CPI and achieve our Level 2 goals in a timely fashion. Process Improvement - Modified Approach In July, 992 Watts Humphrey visited AlS and made a presentation on the Personal Software Process. (AIS is one of three organizations to pilot PSP). In his presentation, Watts made the following key points:. it is practically impossible to produce a usable process the first time. 2. you can not usefully evolve your process definition, until it reasonably represents what you do 3. when your actual, official, and perceived processes converge on your initial process, you will find your process improvements most successful 4. many process definition problems concern seemingly minor details. It is such details, however, that make the difference between an annoying and inconvenient process and a comfortable and efficient one. (Note: At the time of his presentation at AIS, Watts Humphrey was in the process of writing his book A Discipline For Software Engineering I [5] which has since been published by Addison Wesley. Please refer to chapter 3, Defining the Software Process, for a detailed discussion of these concepts.) We realized the significance of these points and were able to relate them to our experience. We could see that while at a high level, we had a sound approach to process improvement, the devil was in the details. Our process definition efforts had two problems:. They were not usable, as written. 2. They described an official process based on our engineers perceptions and represented a 684

5 Proceedings of the 29th Annual Hawaii Inte rnational Conference on System Sciences major departure from the actual process they were using. We had to change our approach to the CPI. In his presentation, Watts described the use of a simple but effective mechanism in PSP, by which engineers improved their personal process. The mechanism was a Process Improvement Proposal (PIP) form for use by engineers to record process improvement ideas as and when they occur to them. I could see that the PIP mechanism had the potential to help us overcome our two major organization tendencies.. If engineers are empowered to gradually evolve AIS processes by means of PIPS, there was a real possibility that they would not look at process definition as an all or nothing proposition. They would know that even if the process is not usable as written, they will have the ability to change it to be usable. 2. PIPS also had the potential to be a practical way to address the minor process details that were annoying and inconvenient. Engineers would use the PIP mechanism to suggest many small incremental changes instead of waiting for few large changes. We decided to implement the PIP mechanism immediatelv. I made the SEPG responsible for the following:. Evaluate PIPS. 2. Provide rapid feedback to the PIP authors regarding the disposition of the PIPS. 3. Maintain PIP records and report PIP status in quarterly status reviews. To highlight the importance of the PIPS and CPI, we moved the SEPG report to the beginning of the quarterly status reviews. The following sections provide quantitative evidence of the remarkable results we achieved as a result of implementing the PIP mechanism. The source of the data is the AIS PIP manuals maintained by the AIS SEPG. I reviewed 67 PIPS and performed the following analysis:. Assigned each PIP to the CMM level it pertained to. 2. Related each PIP to a specific KPA, KPA Goal and KPA Common Feature. The impact of the modified approach using the PIP mechanism is described under the following major topics:. Process maturation and evolution across KPAs 2. Convergence of the official, perceived, and actual processes 3. AIS Defined Process - the initial target process 4. Participative management with discipline Process Maturation and Evolution Across KPAS During , AIS engineers submitted 67 PIPS. Of these 0 were implemented. I was able to assign all but one of the 0 to specific CMM Level and KPA. Table 5 provides the breakdown of the 09 PIPS implemented across CMM Levels and KPAs during the 6 quarters starting in July 992. Table 5. PIPS and CMh4 Levels, KPAs Maturity Key Process Area No. of Ql Q4 Q5 46 Level PIPS Repeatable Requirements Management Software Project Planning Software Project Tracking and Oversight 2 6 Software Quality Assurance Software Configuration Management 3 3 Software Subcontract Management Defined Totals Organization Process Focus Organization Process Definition Training Program Integrated Software Management 3 6

6 Technology Change Management Process Change Management Totals In July 992, when the PIP mechanism was implemented, AIS was a Level organization trying to define processes for moving to Level 2. As can be seen from the table above, during , AIS engineers submitted 49 PIPS pertaining to Level 2 KPAs and 5 PIPS pertaining to Level 3 KPAs. The data clearly indicates that AIS engineers no longer viewed process definition as an all-or-nothing proposition. AIS engineers were defining, and documenting processes across maturity levels and KPAs, even though the organization lacked the maturity to measure, verify and improve the processes. The data also shows more PIPS pertaining to Level 3 KPAs were implemented during the last 3 quarters. Detailed analysis of the PIPS pertaining to Level 2 and Level 3 KPAs follows: Level 2 KPAs Table 6 relates PIPS to the goals and activities of the CMM Vl. Level 2 KPAs. Table 6. PIPS and CMM Level 2 KPAs Cey Process Area Requirements Management Key Practices. System requirements allocated to software are controlled to establish a baseline for software engineering and management use.. The software engineering group reviews the allocated requirements before they are incorporated into the software project.. Software estimates are documented for use in planning and t--- l----- Software Project Planning tracking the software project. 2. Software project activities and commitments are planned and documented. 3. Affected groups and individuals agree to their commitments related to the software project. Ability to perform:. A documented and approved statement of work exists for the software project.. The software engineering group participates on the project proposal team. 5. A software life cycle with pre-defined stages of management size is identified or defined. 6. The project s software development plan is developed according to a documented procedure. 7. The plan for the software project is documented. No. of PIPS

7 9. Estimates for the size of the software work products (or changes to the size of the software work products) are derived according to a documented procedure. 0. Estimates for the software project s effort and costs are derived 0 according to a documented procedure. 2. The project s software schedule is derived according to a 3 documented procedure. 3. The software risks associated with the cost, resource, schedule, 4 and technical aspects of the project are identified, assessed, and documented. 4. Plans for the project s software engineering facilities and support tools are prepared, Measurement and analysis:. Measurements are made and used to determine the status of the software planning activities. Software Project Tracking. Actual results and performances are tracked against the software plans. 6. The project s software effort and costs are tracked and corrective 3 actions are taken as necessary. 2. The software engineering group conducts periodic internal reviews to track technical progress, plans, performance, and issues against the software development plan. 3. Formal reviews to address the accomplishments and results of 7 the software project are conducted at selected project milestones according to a documented procedure. Software Quality Assurance 4. Non-compliance issues that can not be resolved within the software project are addressed by the senior management. 7. Deviations identified in the software activities and software work Software Configuration products are documented and handled according to a documented procedure. 2. Selected software work products are identified, controlled, and Management available. 3. Changes to identified software work products are controlled. 2 Activities Performed: 4. The software work products to be placed under configuration management are identified. 6. Changes to baseline are controlled according to a documented procedure. 2 The data shows that 44 of the 49 Level 2 PIPS pertained to Project Planning and Project Tracking KPAs. This is not surprising, because we were aware that we had to get management control of projects and commitments in order to move out of the Initial Level, AIS engineers used the PIP mechanism to evolve the AIS project plan process. The result was a process that was usable and contained all the tasks and details that was closer to the actual process that the project managers and engineers used to plan projects. Percent of unplanned tasks reported by projects in quarterly status reviews decreased significantly. Project status reports in July 993 quarterly status review showed that 6 687

8 of 7 projects reported no significant schedule deviation. The results of self assessments conducted in January 993 and July 993 showed that we had made significant progress in one year. As can be seen in Table 7, the number of yes answers to TR-23 questions pertaining to Level 2 KPAs increased dramaticaily compared to the results from the January 992 and July 992 self assessments. Project team interviews confirmed that the findings from January 992 did not re-appear as findings. Table 7. Responses to TR-23 - Jan 93 & Jul 93 No. Question Jan 93 Ju93 % yes % yes..3 Does the Software Quality Assurance (SQA) function have a reporting channel separate from development management?..6 Is there a software configuration control function for each project? Is a formal procedure used in the management review of each software development prior to making contract commitments? 2..4 Is a formal procedure used to assure periodic management review of the status of each software nroiect? nf s&ware nmiectc? Level 3 KPAs Table,8 relates PIPS to the goals and activities of the CMM VI. Level 3 KPAs. Table 8. PIPS and CMM Level 3 KPAs Key Process Area Organization Process Definition Key Practices. A standard software process for the organization is developed and maintained. 2. Information related to the use of the organization s standard software process by the software projects is collected, reviewed, and made available. Commitment to perform:. The organization follows a written policy for developing and maintaining a standard software process and related process assets.. The organization s standard software process is developed and maintained according to a documented procedure. 2. The organization s standard software process is documented according to established organization standards. 3. Descriptions of software life cycles that are approved for use by the No. of PIPS

9 Training Program Integrated Software Management Software Product Engineering Intergroup Coordination projects are documented and maintained. 6. A library of software process-related documentation is established and maintained.. Training activities are planned. 2. The organization s training plan is developed and revised according to a documented procedure.. The project s defined software process is a tailored version of the organization s standard software process. 2. The project is planned and managed according to the project s defined software process. Commitment to perform:. The project follows a written organizational policy requiring that the software project be planned and managed using the organization s standard software process and related process assets.. The project s defined software process is developed by tailoring the organization s standard software process according to a documented procedure. 4. The software project is managed in accordance with the project s defined software process.. The software engineering tasks are defined, integrated and consistently performed to produce the software.. Appropriate software engineering methods and tools are integrated into the project s defined software process. 2. The software requirements are developed, maintained, documented, and verified by systematically analyzing the allocated requirements according to the project s defined process. 3. The software design is developed, maintained, and verified, according to the project s defined software process, to accommodate the software requirements and to form the framework for coding. 4. The software code is developed, maintained, and verified, according to the project s defined software process, to implement the software requirements and software design. 5. Software testing is performed according to the project s defined process. 8. The documentation that will be used to operate and maintain the software is developed and maintained according to the project s defined software process. 9. Data on defects identified in peer reviews and testing are collected and analyzed according to the project s defined software process. 2. The commitments between the engineering groups are agreed to by the affected groups. 3. The engineering groups identify, track, and resolve Intergroup issues. 2. Representatives of the project s software engineering group work with representatives of the other engineering groups to monitor and coordinate technical activities and resolve technical issues

10 2. Defects in the software work products are identified and removed. The data shows that 46 of the 5 PIPS related to 3 of the 7 Level 3 KPAs - Organization Process Definition (9) Software Product Engineering (6), and Integrated Software Management ( ). Reviews of the individuals PIPS indicate the following: 3 of the 9 PIPS related to Organization Process Definition resulted in new standards and changes to existing standards covering process elements such as software requirements specifications, cyclic software design, coding, and testing, inspections checklist, test incident reporting and tracking. AIS engineers were creating significant process assets as they matured and evolved Level 2 KPAs. It is also a tribute to the inherent evolutionary structure of the CMM, that the creation of process assets naturally follows once the organization has the foundation of a repeatable process in place. 7 of the 6 PIPS related to Software Product Engineering added significant new practices for software requirements elicitation and analysis. 9 of PIPS related to Integrated Software Management resulted in significant additions to project post mortem practices and enabled the institutionalizing of documenting and storing technical and management lessons learned. In summary, analysis of the PIPS pertaining to Level 3 KPAs indicate that AIS engineers:. Gradually decomposed software process and defined process elements with greater granularity. 2. Incorporated effective methods for requirements elicitation and analysis. 3. Ensured lessons learned are documented systematically. Convergence of the Official, Perceived, and Actual Processes AIS engineers used the PIP mechanism to evolve the official process and bring it closer to the actual process they were using. Analysis of the 0 implemented PIPS indicate the following:. 8 PIPS resulted in additions of tasks to the AIS Work Breakdown Structures. Without such disciplined input from the practitioners, it would have been nearly impossible for SEPG or any staff group to define a process that matches what engineers actually do. Unplanned activities in project plans and actuals decreased. 2. Approximately 20 PIPS can be categorized as clarifications, guidelines, changes to forms, consistency checks, operational definitions etc. These were the minor annoyances and inconveniences mostly resulting from subtle differences between the official and actual processes. These PIPS resulted in small incremental changes. This is a significant development in that the engineers were beginning to see the benefits of many small changes instead of waiting for few large changes. Analysis of the 5 returned PIPS also indicate that the PIP mechanism helped in converging the perceived process with the official and actual processes. About half of the returned PIPS were due to the author s misconception of either the official or actual process and it s intent. With SEPG consultation, the misconceptions were clarified and the need for the PIP eliminated. SQA non-compliance notices declined from an average of 2 per project in 992 to where they are more the exception than normal. The AIS Defined Process - The Initial Target The AIS defined process consists of phases, activities and tasks. Each phase consists of major activities further divided into tasks. Several activities related to project management, project planning, project tracking, software quality assurance, and software configuration management, and software verification & validation occur in each of the phases. As of April, 994 the AIS defined process contained the following: 690

11 * 6 distinct process phases * 73 defined activities further broken down into 504 defined tasks * Of the 73 activities, 40 were unique and 33 were repeated in multiple phases * Of the 504 defined tasks, 358 were unique and 46 were repeated in multiple phases. Majority of the activities and tasks were added as a direct result of the PIPS. AIS engineers have used the PIP mechanism to evolve the AIS defined process and consequently have taken ownership of the defined process. The SEPG role during this time has gradually shifted to a review and facilitating role, PIPS have allowed the practitioners to actively participate in defining the AIS process. Our data supports the view that in Level organizations, starting to define processes is itself the major improvement activity. The defined process coupled with the PIP mechanism will now enable real improvements to the defined process to improve quality, and reduce cycle time. Participative Management With Discipline Participative management and empowerment are currently in vogue. But studies continue to show that in many organizations, such schemes result only in changes to parking privileges and cafeteria menus without producing significant real benefits in core competencies. To quote Watts Humphrey, There is a continuing conflict between directive and participative management. Directive management is often demeaning and demotivating while participative management is often relaxed and permissive. For participative freedom to pay, it must be accompanied by professional discipline. At AIS, the PIP mechanism gave the engineers the professional discipline to successfully participate in process improvement efforts. Engineers had to describe their improvement suggestions in less than 5 lines. They had to convince their peers of the value of their suggestions and learn to accept rejections. SEPG had to use a disciplined process to record, evaluate and act on PIPS with rapid feedback to PIP authors. As shown in Table 9, 9 of the PIPS were about PIPS. (i.e. Process Change Management - a Level 5 KPA). Table 9. PIPS about PIPS Key Process Key Practices No. of PIPS Area Process Change. Continuous process improvement is planned. Management 2. Participation in the organization s software process improvement 7 activities as organization wide. 3. The organization s standard software process and the project s defined processes are improved continuously.. A software process improvement program is established which empowers the members of the organization to improve the processes of the organization. 2. The group responsible for the organization s software process activities 2 (e.g. software engineering process group) coordinates the software process improvement activities. 5. Software process improvement proposals are handled according to a 3 documented procedure. 9. Records of software process improvement activities are maintained. 0. Software managers and technical staff receive feedback on an event- 2 driven basis. Table 0 shows the rate at which PIPS were submitted growing from 2 in 992 3rd quarter to 55 in 9934th quarter. This rate continues to increase. 69

12 Table 0. PIPS across 6 quarters Qtrl Qtr2 Qtr3 Qu4 Qtr5 Q&6 No. of PIPS Table shows the broad participation in PIPS by AIS engineers. Table. PIPS by individual engineers 9 individuals submitted PIPS ranging from to 29 per individual. It is not surprising that the person with 29 PIPS is now the Development Manager at AIS. Also, 9 of the 9 were either Project Managers or have since been promoted to Project Managers. We provide recognition and reinforcement through annual cash awards for the three top PIPS. PIP it has now become the AIS mar&a. Benefits Of Process Improvement During , AIS revenues and profits increased significantly. While not all of the increases can be directly attributed to our process improvement initiative, we believe that the CPI initiative and AIS staff participation in CPI were major contributing factors. 993 was the first full year in which AIS achieved revenue and profit goals. 994 revenues were more than double the 992 revenue. More importantly, profit as a percent of revenue increased from less than % in 992, to 6.55% in 993 and 8.86% in 994. Conclusions. Continuous improvement as a way of life need not wait until organizations acquire ability to measure, verify, and improve processes at higher maturity levels. 2. A simple and effective mechanism can be implemented to enable and guide the gradual evolution and maturation of process definition even in low maturity organizations. 3. Tendency to view process definition as an all-or-nothing proposition can be overcome by empowering practitioners to implement changes and by providing rapid feedback, 4. OBicial, perceived, and actual processes begin to converge when practitioners suggestions are implemented in an institutionalized way. 5. Starting to define processes is itself the major improvement activity in Level organizations. As the AIS process maturity evolves, we hope to be able to report quantitative benefits tied to individual PIPS. Maybe in our next paper. Stay tuned. References. Paulk, M., et al, Capability Maturity Model for Software, Technical Report CMU/SEI- 9 l-tr-24: The Software Engineering Institute, Carnegie-Mellon University, Humphrey, W.S., Managing the Software Process : Addison-Wesley, Humphrey, W.S., et al, A Method for Assessing the Software Engineering Capability of Contractors, Technical Report CMUISEI-87-TR-23: The Software Engineering Institute, Carnegie-Mellon University, September leee, IEEE Standards Collection - Software Engineering: 992 edition. 5. Humphrey, W.S., A Discipline for Sohare Engineering : Addison-Wesley,