Tailoring CMM to a Process Improvement Project

Transcription

1 D. Rubio et al./proc. of Argentine Symposium on Software Engineering (2005) Tailoring CMM to a Process Improvement Project Diego M. Rubio, Patricio A. Maller, Álvaro Ruiz de Mendarozqueta {Diego.Rubio, Patricio.Maller, aruizdemendarozqueta}@motorola.com Motorola Argentina, Centro de software Avda. Hipólito Yrigoyen 146, Piso 9 (X5000JHO) Córdoba, Argentina. Abstract. Management of SPI projects is a key success factor to implement the Capability Maturity Model (CMM ) in an organization. This paper describes how the Motorola s Global Software Group in Cordoba Argentina tailored the organizational process to be used in a process improvement project. All key process areas of the model are discussed, as well as strategy followed. Quantitative results of the experience are also presented, relating the organizational goals with the process improvement project results. Key words. CMM, Capability Maturity Model, process improvement, tailoring. 1. Introduction The CMM model is based in the process management premise, the quality of a system is highly influenced by the quality of the process used to develop and maintain it [1][2]. The different improvement models or cycles, from the Deming wheel to the IDEAL model developed by the SEI [3], have similar structure and steps. The planning and execution of the project for implementing SPI (Software Process Improvement) emerges as essential to all models. Although the SEI (Software Engineering Institute at Carnegie Mellon University) initially sketched guidelines to conduct SPI [4], no clear focus on how to manage these initiatives has been provided. As Humphrey stated [5], if process improvement is not rigorously planned and tracked, it will not happen. Although the CMM model [2] directly addresses the need for a SPI plan, SPI projects often fail due to bad planning [6], [7], to the point of being considered a common mistake in SPI projects. In organizations embracing CMM for managing software projects, it is certainly an opportunity to extend these practices to other domains [8]. Successful experiences using the CMM model in small organizations [9][10] and strong recommendations to apply common sense in their use [11][9] fostered the idea of adapting CMM to a process improvement program. This paper describes the experience of the Global Software Group for Motorola in Cordoba Argentina managing its SPI initiatives by tailoring CMM practices. Currently, the organization operates at CMM level 5, formally assessed in December 2004.

2 CMM Level 142 D. Rubio et al./proc. of Argentine Symposium on Software Engineering (2005) Planned Process Improvement Early adopters of the model have recommended a rigorous approach to process improvement project [12]. More recently, it has been identified as a critical factor [7] [13] remarking the inconsistency of managing process improvement projects as CMM (Capability Maturity Model) level 1 projects [6]. The problem has been identified in the form of organizational patterns, as Process is product and Improvement follows process [14]. The underlying principle behind these patterns is leading by example [15]. Practitioners will be surely reluctant to follow processes not developed using the process they are encouraged to use, and whose quality level is far from those demanded to software projects. It is certainly challenging to tailor the software process to fit a process improvement project [8], but results, as we will show in this paper, are worthy. In the next section, a generic mapping from CMM to a tailorized version of the Organizational Software Process will be presented. Details of tailoring fall beyond the paper, limiting the description to the big components of the process used by the process improvement team. The project described had the goal of defining and deploying the process assets necessary to take an already CMM level 3 organization to operate at a CMM Level 5. Process improvement projects to achieve a CMM Level 3 were also conducted following CMM practices, and therefore, there was a history to allow an informed tailoring of the organizational process. Furthermore, this approach was highlighted as one of the organizational strength by the assessment team, remarking the Strong Project Management Approach to Both Product Development and Process Improvement [16]. 3. P Map to the Process Improvement Project CMM practices embedded in the organizational software development process were tailored to fit the process improvement project as described in the following subsections [17]. 4. P 2 Key Process Areas Requirement Management Establishing and maintaining an agreement with the customer on requirement to be fulfilled is vital for any project to be successful. As stated in the CMM model, the customer may be interpreted as an internal customer as usually happens with Process Improvement projects. At the beginning of this project, a list of requirements was established and documented based on prior assessments findings, process opportunities reports collected by the SEPG and OPG (Organizational Process Group,

3 D. Rubio et al./proc. of Argentine Symposium on Software Engineering (2005) conformed by project s primary customers and sponsors) expectations. These baseline requirements were used as the basis for project estimation and resources allocation creating solid basis upon which commitments could be made by team members. The participation of the customer in the CCB (Change Control Board) of the project was a key factor that enabled the team to assess changes prior to incorporation and also generated a common understanding of the impact of each change. As a consequence, changes were minimized. In addition, practices of requirements traceability, in the form of a standard RTMX (Requirements Traceability Matrix), enabled an accurate analysis of changes impact. Software Project Planning As part of the kick-off meeting, a Project Leader from the development area was allocated to this project. This was a key contributor in managing the Process Improvement project as a regular software project. As required, project plans were initiated as soon as resources were allocated to the project and were documented using a tailored version of the standard project plans. The established agreement on requirements was used to develop a WBS (Work Breakdown Structure) of the product to be generated and standard lifecycle phases were tailored as described in ISM (Integrated Software Management) to meet project necessities. The successful commitment of partially allocated resources was achieved by involving team members in early stages of planning and estimation sessions. Some roles of the software production process were tailored, i.e.: the Technical Leader for the SPI project was a senior quality engineer from the quality department having the responsibility of ensuring a solution based on the requirements. Software Project Tracking and Oversight As this project was considered as a regular project, organizational forums were used to track progress. The project presented its status on weekly meetings where development projects also present its status to the organization. A standard set of slides and metrics was used, ensuring the tracking and reporting of every attribute mandatory in the organizational process (examples of tracked metrics include but are not limited to: size of work product developed, effort, calendar days and milestones, peer review metrics under statistical process control, cost of quality, cost of poor quality and critical computer resources). Project staffing was tracked in detail since the project was planned to fluctuate between 8 and 30 team members. Software Subcontract Management As the decision was to develop all the project activities within the organization using only internal resources (i.e. without consultants participation) this KPA (key process area) was stated as not applicable for this particular project.

4 Level 144 D. Rubio et al./proc. of Argentine Symposium on Software Engineering (2005) Software Quality Assurance A project quality assurance plan was developed using the standard organizational template. The main difficulty in developing and applying SQA (software quality assurance) to this project was the fact that process and quality staff was fully involved in the project; therefore objectivity could be compromised when performing audits. To overcome this situation, some of the most experienced software engineers not participating as team members were selected and trained as internal auditors and acted as project auditors. Standard checklist and plans were followed without major changes. As usual, the noncompliance issues were addressed by the team following the noncompliance resolution process and issues that could not be resolved within the team were escalated to senior management for resolution.. The adoption of a selected SQA group formed by experienced engineers was another interpretation of the model in order to fulfill the objective evaluation of the project activities. Software Configuration Management Due to the fact that a considerable number of assets were planned to be modified and many team members were planned to participate, full discipline of configuration management was applied as in regular software projects. Naming conventions and configuration items identification were exhaustively planned to ensure consistency within team members. Branching schemas were used to enable the evolution of assets already being modified, until merging back project changes. Release management procedure was followed to generate builds prior to publishing assets to ensure system consistency among products integrated for each release. Labeling schemas were defined and applied and status accounting reports were periodically generated. Also, a physical configuration audit was performed before each release in compliance to the organizational process. 5. P 3 Key Process Areas Organizational Process Definition Contribution of this KPA was twofold. First, process, guidelines and templates both at project and product level were extensively used. Templates for project plans, procedures for estimations and reviews, process definition templates, are just a few examples of the level of reuse of software production processes to the improvement project. Second, the Project Historical Database provided metrics on productivity, estimation accuracy and level of defects of previously defined process improvement projects. Also, knowledge sharing from post-mortem meetings enabled rapid risk identification and managing.

5 D. Rubio et al./proc. of Argentine Symposium on Software Engineering (2005) Organization Process Focus This KPA would normally impact only the organizational level, but being a process improvement project, it was much more relevant. Project s CCB was formed by the complete SEPG (software engineering process group) and senior management. Different champions from SEPG (e.g. metrics, defect prevention) actively participated on requirements definition and development. Commitment to informal assessments in a quarterly basis was managed as a testing requirement for the project. Results of assessments, after CCB revision, were incorporated into the project s requirements. Informal assessments were led by experts external to the organization, who remarked the use of process improvement project plan as an instance of the organizational process improvement plan. Training Program The project maintained a specific training plan, containing the minimum set of skills required to project team members. The Organizational Training Plan was a workproduct of the project in order to include higher maturity KPAs required trainings. Integrated Software Management The spirit of this paper is directly related to goal 1 of this KPA: the tailoring of the organizational process to fit specific project needs. Life cycle, process modifications and templates modified were explicitly declared in project plans. Process compliance was evaluated through SQA audits, and a very productive peer revision in regular project reviews. Reviews showed that thresholds defined to manage deviations in effort or calendar in software projects were also appropriate for process improvement projects. Moreover, solutions for managing these deviations were also analogous. Risk management and issue tracking activities were also conducted as in regular projects. Software Product Engineering At a glance this KPA should be the least applicable to a process improvement project. However some of the main components were considered and used. A phased development life cycle, with regular quality gates was defined for every workproduct. Pilots were the choice of testing for processes and guidelines. Traceability of workproducts was maintained using the standard RTMX of the organizational process. Functional configuration audits were performed in compliance with the organizational audit process.

6 Level 146 D. Rubio et al./proc. of Argentine Symposium on Software Engineering (2005) Intergroup Coordination Since the project run on marginal allocation of many members, intergroup dependencies with projects was carefully managed. Issues were conveniently escalated in project reviews. Moreover software project schedules were asked to reflect the allocation of the process improvement team members. Senior management sponsorship and participation was a critical success factor. Peer reviews Peer reviews were embedded in the project by design. Every written document followed the organization standard peer review process. Peer reviews were included in the process improvement project schedule. Being peer reviews a very institutionalized practice, team members were well aware of the usage and importance of defect classification and metrics. Release audits verified that plan was correctly implemented, using objective evidence in each case. 6. P 4 Key Process Areas Quantitative Process Management and Software Quality Management The overall strategy for the CMM level 4 was to managed both KPAs at the organizational level and project level depending on the granularity of the metrics and the goals level, (organizational or project). For instance, inspection data and goals is tracked at project level and in project reviews, and the overall cost of quality of projects is tracked at the organizational level with the scorecards goals. Historical data regarding basic metrics for SPI projects was available and also a rich historical set of data was available for document inspections and technical reviews. Once level 4 KPA strategy was developed, to pilot assets and practices on this project was decided. This decision enabled the project to collect early feedback on implementation without impacting open software projects and gave the organization the intangible value of leading by example. At the beginning, peer review metrics were selected as to be tracked using statistical techniques. Metrics were designed and developed in order to cover both product (Software Quality Management) and process (Quantitative Process Management) attributes (including productivity, effort effectiveness and fault density). Control charts were reviewed on weekly basis in the project tracking meeting enabling every member of the organization to be exposed to such practices. In fact, this mean turn out to be a very important contribution to level 4 related practices institutionalization. Finally, quantitative goals were updated and prioritized based on organizational baseline of project s applicable metrics. Tracking of those metrics against control and specification limits were included in weekly reports in order to proactively act

7 Level D. Rubio et al./proc. of Argentine Symposium on Software Engineering (2005) towards project goals achievement. Whenever a special cause of variation was detected, causal analysis meetings were held in order to address the root cause of this variation. 7. P 5 Key Process Areas Process Change Management and Technology Change Management Some of the non-functional requirements for the process improvement project concerned the automation of many process steps and metrics collections. To achieve these goals, many processes had to be modified or redefined, and tools had to replace manual procedures. The approach was then to follow an iterative path to maturity. With the metrics at hand at the moment, and the existing processes, a process performance baseline was generated. A pilot was executed for the process or tool in selected low risk projects and results evaluated, when possible, re-calculating a process capability baseline. Based on the quantitative data, the decision on the massive deployment of the process and tools was finally made. The iterative approach implied that the first technology change management plan, or pilot report, were constructed without a template, but once the process improvement wheel was spinning, a full set of assets was adopted. Defect prevention The process improvement project was a fertile field to pilot all defect prevention practices. Although the level of effort traced to poor quality in process improvement workproducts was minimal, the effort devoted to research was very likely to be improved. The use of causal analysis meetings, geared with quality tools such as Pareto charts, Ishikawa diagrams and correlation analysis was introduced by the project and rapidly adopted by the rest of the organization. 8. Results The success of the SPI project, in contrast with other SPI initiatives, was not only evaluated by the success of the formal assessment, but also by other factors that would be normally required for software projects: slippage in effort and schedule estimations, cost of quality and poor quality, and practitioners satisfaction with the implementation. Quality of workproducts was evaluated by a formal SEI CBA/IPI assessment team (4 external members, 3 of them authorized SEI CMM lead assessors, and 2 internal). The assessment revealed 8 improvement opportunities in 3 different KPAs, and highlighted as organizational strengths the commitment to process improvement

8 148 D. Rubio et al./proc. of Argentine Symposium on Software Engineering (2005) and the high integration of tools. Additionally, CMM philosophy indicating that the quality of the process followed is a determinant of product quality was corroborated by projects operating at level 5, which consistently fulfilled the organizationally established goals on costs and quality. An initial gap analysis with CMMI was also developed as part of the assessment. CMM Level 5 of operation was achieved in less than 18 months from the CMM Level 3 formal assessment, about a half of the average reported to the SEI [18]. However, in times where lean and agile are more than buzzwords in the market, achieve a high maturity profile no matter the cost was not an option. SPI project goals on costs and schedule were aligned with the Organizational goals as recommended by CMMI (Capability Maturity Model Integration) [19][20]. Overall, the project consumed 6400 staff-hours, with an average of 8 team members and peaks of 40. Project duration was 260 days. In any other context, running a project of this size without rigorous project management techniques would not have been recommended [8]. In process improvement projects, however, informality is not a rare example. One of the most critical factors was then to manage deviations of schedule and estimated effort. More than 62 milestones were defined, and weekly tracking meetings established. Issues such as resources shortage and delayed activities were frequently reported to and addressed by Sr. Management. Project finished on time, and with a deviation on the last estimate of effort below 15%. Another cost item efficiently managed was changes to original requirements, limited to 8 along the life cycle. Having stakeholders in the project CCB was absolutely crucial to keep changes at a minimum and impact analysis accurate. Similarly to software projects the focus on how much appraisal and rework was spent by the project was present during the whole life of the project. In more than 100 peer reviews, around 300 errors were found and corrected. Overall, the cost of quality of the project was around 13%, with a cost of poor quality component of 3%. Peer review process was managed using statistical process control, providing early feedback on defect density, and overall effectiveness of the review. Finally, satisfaction of practitioners with process improvement results was assessed by the number of changes requested on deployed tools and processes. Six months after the closing of the process improvement project, only 4 proposals were received, none of them critical. 9. Conclusions Management of SPI projects is a key success factor to implement CMM in an organization. This paper describes the approach adopted by Motorola s Global Software Group in Cordoba Argentina. A mapping of CMM practices to SPI projects components was presented as used in the effort that led the organization to achieve a CMM level 5 status. Although descriptions are general, an initial tailoring to be followed by other initiatives was presented.

9 D. Rubio et al./proc. of Argentine Symposium on Software Engineering (2005) Results of the experience were encouraging, achieving a CMM maturity level 5 in half of the average time, still complying with costs and staffing restrictions. A number of indicators point out CMM practices as a whole as the success factor, including project and process management, engineering and support KPAs. As a consequence, this approach has been adopted as the standard for SPI projects. 8. Further Research Further work is necessary to formalize a rigorous approach to process improvement in such a way that different organizations may follow it. However, the general bootstrap strategy of adopting the reference model as guide has been clearly suitable for the organization. Since CMM is in a sunset, verifying the approach including CMMI practices presents a new challenge. Few examples of CMMI implementations in non traditional projects, such as SPI, are currently available. No significant variations are expected in the practices described in this paper, although a full process improvement cycle in a CMMI environment will be required to verify the approach. 10. Bibliography 1. Paulk Mark et all, Capability Maturity Model for Software, Version 1.1, Software Engineering Institute Carnegie Mellon University Pittsburgh, Pennsylvania 15213; February 1993 CMU/SEI-93-TR-24 ESC-TR Paulk Mark et all, Key Practices of the Capability Maturity Model, Version 1.1, Software Engineering Institute Carnegie Mellon University Pittsburgh, Pennsylvania 15213; February 1993 CMU/SEI-93-TR-25 ESC-TR Jennifer Gremba and Chuck Myers, The IDEAL Model: A Practical Guide for Improvement, Software Engineering Institute (SEI) publication, Bridge, issue three, Fowler P., Rifkin S., Card D.; Software engineering process group guide; Technical Report CMU/SEI-90-TR-024 ESD-90-TR-225; Software Engineering Institute. Carnegie Mellon University, Humphrey Watts, Managing the Software Process, Addison-Wesley Longman Publishing Co., Inc., Boston, MA, Hefner R., Tauser J., Things They Never Taught You in CMM School Software Engineering Workshop, Proceedings. 26th, Rainer A., Hall T., Campus H.; Key success factors for implementing software process improvement: A maturity-based analysis; Journal of Systems and Software, Johnson, D.L.; Brodman, J.G.; Applying CMM project planning practices to diverse environments; Software, IEEE, Volume 17, Issue 4, July-Aug Page(s): Paulk Mark, Using the Software CMM in Small Organizations, Software Engineering Institute Carnegie Mellon University Pittsburgh, Available by request from the Software Engineering Institute, Carnegie Mellon University, Donna L. Johnson, Judith G. Brodman, Tailoring The CMM for Small Businesses, Small Organizations, and Small Projects, Software Process, Newsletter. IEEE Computer Society, No. 8, Winter 1997, pp. 1-7.

10 150 D. Rubio et al./proc. of Argentine Symposium on Software Engineering (2005) Paulk Mark, Using the Software CMM With Good Judgment, ASQ Software Quality Professional, Vol. 1, No. 3, June 1999, pp Caputo Kim, CMM Implementation Guide: Choreographing Software Process Improvement, Addison-Wesley Professional, Fitzgerald B., T O'Kane T.; A Longitudinal Study of Software Process Improvement; IEEE Software, IEEE Volume 16, Issue 3, May-June 1999 Page(s): Appleton B.; Patterns for Conducting Process Improvement, Washington University Technical Report# WUCS-97-34, PLoP Technical Proceedings; Juran J. M., Juran y El Liderazgo para la Calidad, Díaz de Santos, Madrid, SEI Appraisal Program at SEI ( 17. Ginsberg Mark, Process Tailoring and the Software Capability Maturity Model, Software Engineering Institute Carnegie Mellon University Pittsburgh, Pennsylvania 15213; Technical Report CMU/SEI-94-TR-024 ESC-TR Software Engineering Institute. Carnegie Mellon University ; SW-CMM Maturity Profile Chrissis M. B., Konrad M., Shrum S., CMMI: guidelines for process integration and product improvement; Addison-Wesley Longman Publishing Co., Inc., Boston, MA, CMMI Product Team, Capability Maturity Model Integration (CMMISM), Version 1.1, Pittsburgh, Pennsylvania 15213, 2002.