Methodological Issues in a CMM Level 4 Implementation

Transcription

1 Methodological Issues in a CMM Level 4 Implementation Giuliano Antoniol, Sara Gradara, Gabriele Venturi antoniol@ieee.org, gradara@unisannio.it, venturi@unisannio.it, RCOST - Research Centre on Software Technology University of Sannio, Palazzo ex Poste via Traiano 1, Benevento, Italy Abstract The Capability Maturity Model (CMM) developed by the Software Engineering Institute is an improvement paradigm. It provides a framework for assessing the maturity of software processes on a five level scale, and guidelines which help to improve software process and artifact quality. Moving towards CMM Level 4 and Level 5, is a very demanding task even for large software companies already accustomed to the CMM and ISO certifications. It requires, for example, quality monitoring, control, feedback, and process optimization. In fact, going beyond CMM Level 3 requires a radical change in the way projects are carried out and managed. It involves quantitative and statistical techniques to control software processes and quality, and it entails substantial changes in the way the organization approaches software life cycle activities. In this paper we describe the process changes, adaptation, integration and tailoring, and we report lessons learned while preparing an Italian solution centre of EDS for the Level 4 internal assessment. The solution centre has about 350 people and carries out about 40 software development and maintenance projects each year. We describe how Level 4 Key Process Areas have been implemented building a methodological framework which leverages both existing available methodologies and practices already in place (e.g., derived form ISO compliance). We discuss how methodologies have been adapted to the company s internal and external situation and what are the underlining assumptions for the methodology adaptation. Furthermore

2 we discuss cultural and organizational changes required to obtain a CMM Level 4 certification. The steps and the process improvement we have carried out, and the challenges we have faced were most likely those whith the highest risk and cost driving factor common to all organizations aiming at achieving CMM Level 4. Keywords: CMM, GQM, process improvement, statistical control, software maintenance 1. Introduction The term Software Process Improvement (SPI) groups together all those activities aiming at improving processes in software organizations. Software organizations are more and more investing resources in SPI programs to keep their processes under control; to gain the ability to improve their effectiveness and efficiency; and to gain a better acceptance from important customers. Very often SPI activities are based on the CMM paradigm (see (Paulk et al., 1995) and related documents). The CMM was developed by the Software Engineering Institute (SEI) at Carnegie Mellon University (CMU) to provide software organizations with a paradigm to assess and improve the development and maintenance processes in software organizations. The CMM classifies organizations into five maturity levels: each level is characterized by Key Process Areas (KPAs); each KPA groups together one or more key practices. Increasing the maturity level involves the organization s ability to address the key practices of each KPA of the target level. The CMM defines how to evaluate the organization level but it is not prescriptive with regard to how any particular key practice should be implemented. This has traditionally been a source of criticism; however, it also leaves certain degrees of freedom for implementing the CMM KPAs in the most suitable way depending on each particular context. The CMM aims at helping a very large and differentiated group of companies and organizations. It is focused on the basics ideas, points and areas that can be common to all software organizations. The CMM does not deal with implementation details and practices, it only deals with the desired results. Moreover, process improvement models, such as the CMM, have been accused of neglecting maintenance activities (Hall et al., 2001). While it is true that the CMM does not make explicit references to maintenance activities, this does not prevent the application of the CMM to maintenance organizations. The CMM focuses on high level issues. Peculiarities differentiating development from maintenance are not really essential to the improvement process. The model generality makes it customizable and adaptable to almost all the situations in which

3 the focus of the activity is software. Hence, tailoring it opportunely, the model can be fruitfully adapted to maintenance activities too. One of the purposes of this paper is to report the tailoring activities we carried out in helping a CMM Level 3 software maintenance organization to reach the Level 4. EDS Italia Software is the first Solution Centre (SC) established by EDS, Electronic Data System, in Italy. Its core business is mainly in the area of software maintenance. While figures change from year to year it can be estimated that about 90% of the SC activities are devoted to maintenance. EDS SC has adopted the CMM as a reference model to achieve a continuous process improvement; it has already obtained the CMM Level 3 and ISO certification, and, at the time of writing, it is working toward the Level 4 maturity rating. It is worth noting that no Italian organization is rated at CMM Level 4 or higher, hence this is a very new activity in the Italian software landscape. In this paper we present how EDS SC implemented the CMM Level 4 KPAs, the process tailoring, modification and adaptation in particular with regard to the approaches followed for each key practice, the activities involved, and the changes required in procedures, organization, structure, and culture. As the CMM does not deal with implementation details, implementing CMM Level 4 has been a complex process of methodological tailoring. Other more low level methodologies and approaches have been chosen and adapted to pursue the CMM goals. Furthermore the whole implementation poses interesting organizational and cultural challenges that have been addressed and reported in the paper. The CMM is mainly adopted for two reasons: customers (e.g., organizations such as the USA defense agency) often require some level of CMM compliance; on the other hand, the model is recognized as an effective one, capable of delivering real benefits to the implementing organizations. Although the real extent of the obtainable benefits is under debate, there are several publications (Haley, 1996, Herbsleb et al., 1997, Hersleb & Goldenson, 1996, Humphrey et al., 1991) supporting the effectiveness of the model. In this paper our aim is not to quantitatively evaluate the obtained results in terms of costs/benefits. Hence, we will not devote any particular attention to the economical aspects of this CMM based SPI. In our opinion any conclusion regarding the economic effectiveness of any process improvement practice requires a very large amount of data and the analysis of several years of activities. Instead, we believe that the methodological and organizational aspects of the activity carried out in EDS SC can have an interest beyond the particular organization. Therefore we focus on aspects that stem from this particular CMM Level 4 implementation but are likely to extend their value to other situations. In particular we examine how CMM

4 has been integrated; how other well known methodologies have been adapted to CMM requirements; how existing approaches and procedures have been considered and modified; and the organizational impact of the resulting methodologies. Therefore we present an in depth view of the main methodological, organizational and cultural aspects of CMM Level 4 implementation; we discuss and elaborate the particular choices that have been made and the underlying motivation. The gathered observations can lead to insight and lessons learned pertaining to three areas of interest: a) The structure of CMM Level 4, its strengths and its weaknesses; b) How methodologies can be integrated and modified to cope with pre-existing approaches and new needs; and c) What challenges an organization faces when it addresses important cultural changes such as those required by CMM Level 4. To illustrate these points we summarize actions taken, activities carried out, encountered problems, and solutions adopted. Moreover we describe how the methodological tailoring work has been carried out making particular assumptions regarding the EDS SC s competitive environment, the SC s position in the whole company, its capabilities, and CMM Level 4 goals. A key point of this paper is to make explicit these assumptions and to discuss their impact on the activities carried out; whether these assumptions are compatible with CMM aims; and what can be the effect of these assumptions on the resulting methodologies. We believe that the reached conclusions, although not definitive and open to debate, can be useful to other organizations and of interest to the scientific community. This paper is based on the experiences and information collected by the authors during nineteen months of activities in preparing the organization for its Level 4 internal assessment. Beside the KPAs implementation, during this period numerous activities such as training, auditing and informal interviews were carried out to assess the ongoing process. The remainder of the paper is organized as follows: Section 2 briefly presents the structure of CMM Level 3 and Level 4 and other methodologies relevant for this work; this section simply constitutes a background section on: the CMM, the Quality Functional Deployment, the Goal-Driven Measurement Process, and the Goal Question Metrics. Section 3 presents how the above-mentioned methodologies have been applied by EDS SC for its particular needs and purposes, the tailoring done, and how the resulting methodologies have been integrated. Section 4 presents the process followed to implement the methodologies and focuses on

5 the main cultural and organizational changes. Finally, Section 5 reports effort, costs, and benefits of the improvement activities while in Section 6 we draw some conclusions and observations. 2. Methodological frameworks The CMM is a model derived from the experiences of several software managers and practitioners. It permits different organizations to assess their capability and maturity level. Since organizations addressed by the CMM are very different, the model does not impose implementation details; different organizations are free to select and customize methodologies, models, and approaches to achieve the CMM goals. In our case, EDS SC is pursuing CMM Level 4 goals using the Quality Functional Deployment (QFD) (Mizuno & Akao, 1994) and the Goal-Driven Measurement Process (GDMP) (Park et al., 1996) based on the Goal Question Metrics (GQM) paradigm introduced and described by Basili (Basili et al., 1994). In this section we briefly summarize the key elements of these methodologies with a preliminary comparison between CMM Level 3 and Level From CMM Level 3 to CMM Level 4 Each level of the CMM is focused on a bundle of strictly related capabilities characterizing the particular level of maturity of an organization. The model decomposes each level in different KPAs that are the organizational processes on which capabilities are developed (Paulk et al., 1995). Each KPA is described in terms of key practices (Paulk et al., 1993) contributing to the achievement of its goals. It is in the key practices that the model is open to interpretation and customization. For each key practice the model defines the goals but not how to achieve them. Level 3, the Defined level, is focused on the explicit knowledge of the organization s processes, the use of software engineering practices, and the organic collection of product/process metrics. Level 3 KPAs are: Organization Process Focus, Organization Process Definition, Training Program, Integrated Software Management, Software Products Engineering, Intergroup Coordination, and Peer Reviews. In a typical Level 3 organization there is a Software Engineering Process Group (SEPG) responsible for the engineering practices in use, that is to say the set of procedures for the software processes, and a database of metrics which stores in a consistent way the product and process metrics collected by the organization.

6 Level 4, the Managed level, is focused on product and process quality. It aims at aligning organization quality goals with market requests, establishing a quantitative understanding of software processes and products, and at controlling special causes of variation. Table 1 reports Level 4 KPAs and relative goals (Paulk et al., 1995). KPA Goals 1)The project s software quality management activities are planned Software Quality Management 2)Measurable goals for software product quality and their priorities are defined 3)Actual progress toward achieving the quality goals for the software products is quantified and managed 1)The quantitative process management activities are planned Quantitative Process Management 2)The process performance of the project s defined software process is controlled quantitatively 3)The process capability of the organization s standard software process is known in quantitative terms Table 1. Level 4 KPAs and goals The Software Quality Management KPA s main purpose is to implement a methodology which permits the definition of the quality level of each project in a satisfactory way for both the organization and the customers involved. This quality level must be defined with reference to software metrics collected by the organization. The Quantitative Process Management KPA s purpose is to control the process performance of software projects in a quantitative way. Special causes of variation must be identified and corrective actions must be taken. It is worth noting that CMM Level 3 substantially differs both in purposes and methodologies from Level 4: the former is focused on organizational aspects, software processes, and engineering issues. The organization must use software engineering practices and must standardize software processes. Level 4 key issues are broader: the organization must be able to compare its quality definition with customers needs and to promptly act to improve its processes during their life. In other words, post mortem analysis no longer suffices.

7 To achieve a Level 4 assessment an organization must deal with the following points: Identification of relevant customer and organization goals; Identification of important processes which need to be controlled in order to achieve the previous mentioned goals; Implementation of measures of process and product in order to control the achievement of the previous points; Diffusion in the organization of the needed culture; and Building operative mechanisms to integrate the necessary practices in the organization Quality Functional Deployment CMM Level 4 requires an understanding of the software quality needs and priorities of the customer, and tracing these to software requirements and quality goals. The QFD, a quality assurance technique developed in the 1960s by Shigeru Mizuno and Yoji Akao (Mizuno & Akao, 1994), is useful for recognizing customer needs and priorities; these are often referred to as the Voice Of the Customer (VOC). This is possible through extensive market research, surveys, interviews, feedback mechanisms, and functional and non functional requirements. A clear comprehension of the VOC will lead the organization to focus on customer satisfaction. The QFD provides a system for designing and analysing the deployed quality involving the House of Quality (HOQ) (Hauser & Clausing, 1988). The HOQ is a methodology which permits matching prioritized customer needs with organization abilities and convenience. The HOQ (see Figure 1) is composed of six parts: Customer Requirements, the definition of the VOC. The Customer Requirements are reported on the left hand side of the diagram with an assigned rank (from most desired to least desired). Technical/Design Requirements, the upper section of the house, are those design aspects and technical features which address each of the customer requirements. Planning Matrix, on the right hand side of the house, compares, from the customer point of view, the organization and competitors abilities to meet customer requirements. This is obtained combining customer rating on requirements plus organization and competitor abilities.

8 Technical Correlation Matrix Technical/Design Requirements Customer Requirements Planning Matrix / Customer Perception Interrelationship Matrix Targets: Prioritized Requirements Competitive Benchmarks Technical Target Figure 1. The House of Quality Interrelationship Matrix, the centre of the house, reports the rated connections between customer requirements and technical requirements based on the customer views of the Planning Matrix. Technical Correlation Matrix, the roof of the house, rates the interactions between different technical requirements and identifies eventual conflicts between requirements. Targets, the bottom of the house, summarizes the conclusions following the data contained in the entire house. This part provides values of Prioritized Requirements, and Competitive Benchmarks from the comparison with different competitors, and Technical targets based on Technical characteristics to be improved Goal-Driven Measurement Process and Goal Question Metrics The GQM paradigm is useful for deciding what should be measured to address the organization s defined goals. The GQM methodology presents three levels as shown in Figure 2. Applying GQM requires identifying a list of goals and, for each goal, some questions are identified. The answers to the questions determine if the goals are being met. Then, each question is analyzed to determine what to measure in order to answer it. GDMP extends GQM as described by Basili (Basili et al., 1994) with an intermediate step which helps in linking together questions and measurement results. The GQM paradigm is thus changed into GQ(I)M where I stands for Indicator that is the displays used to communicate the data. Moreover, in GQ(I)M M stands for Measure, not for Metrics. This is due to the fact that the term Measure is an accepted definition while Metrics is a debated term.

9 Goal Definition Question Q1 Q2 Q3 Q4 implicit models Implementation Metric M1 M2 M3 M4 M5 M6 M7 Figure 2. Goal Question Metrics GDMP is based on three precepts (Park et al., 1996): Measurement goals are derived from business goals. The goal driven process begins with identifying business goals and breaking them down into manageable sub-goals; Evolving mental models provide context and focus. The primary mechanisms for translating goals into issues, questions, and measures are the mental models of the processes in use; and GQ(I)M translates informal goals into executable measurement structures. The process ends with a plan for implementing measures and indicators supporting goals. The whole GDMP consists of 10 steps whose analysis is reported in Subsection 3.4 compared with the steps performed in the EDS SC adapted process. 3. EDS interpretation of the methodological framework In CMM Level 4, measures are the base for the decisional processes. The whole Level 4 activity leit-motif is what measures should be taken and how these measures should be used to make decisions. However the CMM is not prescriptive. It does not indicate which particular goals must be pursued, how these can be represented and measured, and how measures must be carried out. EDS SC has filled this gap using GQ(I)M and QFD to implement the CMM Level 4 KPAs goals. Nevertheless, important tailoring has been done on these methodologies to obtain a well integrated approach for achieving the desired benefits in an efficient

10 and streamlinened way. Figure 3 shows the relationship between methodologies used by EDS SC. To address the Software Quality Management KPA a methodology derived from QFD is used. For the Quantitative Process Management KPA a modified version of GDMP has been implemented. In both KPAs, Statistical Process Control (SPC) plays an important role. CMM Level 4 KPA Software Quality Management Quality Functional Deployment (Simplified House of Quality) Statistical Process Control KPA Quantitative Process Management Goal Driven Measurement Processing (Goal Questio-Indicators Measure) Figure 3. Methodological framework. In this section we briefly present the EDS SC organization, we introduce SPC; then we describe how the two KPAs of CMM Level 4 have been implemented by means of QFD, GDMP and SPC, the methodological tailoring, the underlying purposes of the tailoring, and the obtained results. Each methodological subsection is concluded with some observations and preliminary conclusions EDS Solution Center EDS Italia Software located in Caserta is the first Solution Centre setup by EDS in Italy. The organization employs about 350 people including developers, sales staff, managers, and other employees. It carries out about 40 software development and maintenance projects each year. The EDS SC improvement activities began in 1999 under guidelines issued by the headquarters aiming toward the service excellence of the Global Solution Management System (GSMS). The GSMS is the global process used by EDS to best address client needs and expectations.

11 At the time of writing EDS SC is a CMM Level 3 organization and, since 1996, it has been an ISO 9000 (ISO9000, 1998) certified organization. The ISO 9000 is a standards series widely accepted and required in the European market which has in common with CMM the concern for quality and process management. Although similarities can be found (Paulk, 1995), ISO and CMM are different: ISO 9001, the standard more related to software development and maintenance in the 9000 series, deals with minimum requirements for a quality system while CMM emphasizes a continuous process improvement in a more explicit way. The newer version of the ISO standard addresses more directly continuous improvement related topics. However, we refer here to the old version, as it is the one implemented in EDS SC. In 2001 the EDS SC management decided to further increase its maturity level moving up from CMM Level 3 to Level 4 and hence to participate in an improvement plan involving the whole company organization Statistical Process Control and Process Capability Baseline A key capability for CMM Level 4 is the ability to perceive that something in a process is no longer under control, and to promptly take the appropriate actions bringing the process back under control. In addition, CMM Level 4 prescribes that project management activities must be based on quantitative methods. SPC relies on the adoption of statistical techniques and tools to collect and analyze data, and to study and monitor process capability and performance. There are several tools used in SPC that can be grouped in two areas: process control and cause analysis. The purpose of process control tools is the surveillance of processes. These tools allow the organization to detect when a process is under control or when it deviates. Control charts are one of the most used tools to identify undesirable variation caused by anomalies in the process. A control chart displays certain descriptive statistics for a specific quantitative measurement of the process, and compares the actual values with their in control sampling distributions. When a process is under statistical control, that is all measurements fall within their natural process limits, and continue to remain in the same state, the prediction of its future performance is feasible. Since organizations at CMM Level 4 must be able to predict process and product quality trends within measurable limits, they are an elected field for applying SPC process control tools. In EDS SC the Process Capability is defined with a 4 step process: Collection of historical data;

12 Use of Control Charts (mainly XmR Charts) to analyze the collected data, and definition of the organization limits; Consolidation of control limits; and Definition of the Process Capability Baseline. Consolidation of control limits is obtained when a given number of observations is collected and they can be considered representative of the voice of the process related to the considered indicator. When control limits are consolidated, the control chart represents graphically the organization s capability relative to the reported indicators. Therefore it constitutes a Process Capability Baseline. The control charts of the defined indicators are updated periodically verifying the stability of the process. Every six months the capability indexes obtained from the projects, based on the Project Baseline Capability, are compared with the target values associated with each indicator to assess and verify project performance with respect to the organization s limits. When a process is out of control, the second group of SPC tools, the cause analysis tools, are used to determine the variation causes. Once causes are identified, they can be removed and the process is brought back under control. As summarized above, the use of SPC in EDS SC is a fairly classic one; nevertheless some interesting observations can be made. SPC techniques provide a well defined procedure aimed at checking processes trends and variation causes; unfortunately, they do not provide insight on how causes of under performance or over performance can be removed. To bring the process back under control, the EDS SC project managers can use the body of practices of the Organization Standard Software Process (OSSP), and the engineering practices promoted by the SEPG. Nevertheless both cause identification and cause removal activities can often be very difficult and problematic. Variation causes in software projects are frequently tied to people behavior, in other words bringing back under control a process entails working on people and organization. As we will discuss in Section 6 this is a very central point of the whole CMM Level 4 implementation. The methodology chosen by EDS SC to compute project baselines is auto referential: the project baseline is defined starting from the project s first months of activity. This fact has some important implications: the initial activities on a project are not necessarily controllable; it is necessary to wait until a project shows some degree of stability in its statistics, before drawing some conclusions about its control; and each project

13 has very different statistical outfits. Other organizations such as Motorola (Diaz & Sligo, 1997) assign baselines to projects on an analogy based paradigm (i.e., they rely on similar projects carried out in the past). Clearly, this approach offers the advantages of supplying a baseline from the initial stages of the project and of promoting inter project comparison. Nevertheless, we think that given its situation, EDS SC s choice is appropriate. It is worth underlying that the EDS SC s core business is mainly in software maintenance of very large legacy systems; a typical customer is a large financial or insurance firm. Each project can last several years and involves de facto quite different customer goals, required practices and collected software metrics, or different constraints and budget limits. So each project can be very different from others. Therefore, it would be dangerously misleading to adopt of a baseline derived from other projects. Adopting the baseline of a previous project could also lead to negative side effects, people may judge non objective the imposed limits, leading to further organizational problems. EDS SC correctly perceives SPC as an essential issue for the organization s capabilities. Furthermore, a very relevant effort has been devoted and is planned for the near future; in particular, a road map has been implemented in collaboration with the University of Sannio to link each part of the software process of EDS SC (as indicated in the Global Solution Management System) with the statistic methodologies involved in the SPC. The aim of the road map is to help people with regard to the methodologies application: it provides the guidelines for applying the right techniques based on available and needed information. Thus, it is necessary to train people not only on SPC techniques, but also on how to interpret and to apply the road map associated with SPC activities Software Quality Management KPA In the Software Quality Management KPA the organization must understand the quality of the software products, and define and achieve specific quality goals. The Simplified QFD (SQFD) method, proposed by the Software Productivity Consortium and showed in Figure 4, has been adopted to define quality goals. However it must be integrated with further steps to link QFD activities with the other related CMM Level 4 activities. The SQFD is implemented in the first 4 steps of the whole process followed by EDS SC. The process steps are listed in Table 2.

14 Figure 4. Simplified QFD Process step S Q F D 1) Identify customer s Software Quality Needs 2) Quality characteristic and ranking 3) Determining the process voice and the product features 4) Evaluating the relationship values matrix between quality goals and process/product features 5) Definition of quality indicators 6) Complement of project quality plan 7) Project s software products are measured and analyzed 8) Analysis and diagnosis Table 2. Software Quality Management implementation steps. Step 1 corresponds to the Customer Requirements section of the HOQ (see Figure 1) and involves the understanding of the software quality needs of the customer. To elicit the customer needs (i.e., the VOC), surveys, interviews, and market researches are used together with the content of the Service Level Agreements between EDS SC and customers. Results are detailed as specific requirements and quantitative project characteristics. In Step 2 quality attribute types are derived from the VOC using the ISO/IEC 9126 FDIS (ISO/IEC9126, 2000) and then ranked for both the organization and the customer. The overall importance of each attribute is determined as the maximum factor of relative importance between the customer and the organization relative importance.

15 Step 3 aims at understanding the project voice, that is the Technical Requirements of the HOQ. The project voice is a list of Process and Product Features addressing customer requirements, identified using the organization standard process, the Project s Defined Software Process (PDSP). In Step 4 the relationship matrix between customer requirements and project needs is evaluated. The result is a numeric indication, for each feature, of the overall impact on the requirements. Important requirements correspond to an elevated importance of the features that fulfill them. The first four steps complete the application of SQFD. Steps 5 to 8 aim at defining and using quality indicators to quantify the goals. Then, for each measurable characteristic of software product quality, a target value is selected as a quality goal: in Step 5 indicators are selected, then in Step 6 the project plan is completed with quality related activities, and in Step 7 and 8 measures are taken and analyzed. In the last two steps, 7 and 8, SPC is used. The whole approach built by EDS SC is subdivisible in three macro phases: From Step 1 to Step 4 we have a fairly classic quality approach aiming at identifying quality issues and representing them as product and process features; Steps 5 to 7 aim at linking product and process features to metrics, and planning metrics collection; and Step 8 consists of process and product auditing, control, and management. All these steps, as implemented in EDS SC, required tailoring. The original version of the SQFD is based on the HOQ paradigm. EDS SC chose to use a simplified version of the paradigm known as the Simplified House Of Quality (SHOQ). The main difference between the two approaches is that SHOQ does not take into account the customer perceived quality neither for the organization nor for the competitors. The same simplification also has an impact on the bottom part of the HOQ where the original version supplies a qualitative benchmark with the competitors while the SHOQ does not. The HOQ entails directly collecting customer opinion on the perceived quality level regarding both product and process features. Furthermore the HOQ entails collecting the opinion of the customer regarding his perceived quality of competitors on the same points. As a result the model permits managing the quality taking into account a quality level that can maximize the quality perceived by customers in a given market situation instead of a predefined standard of quality. Since quality has a cost, the whole HOQ is aimed at

16 deploying exactly the required quality level. Clearly, the exact amount of quality must be determined with respect, among other factors, to the quality deployed by competitors. The simplifications taken by SHOQ made it relatively lightweight with respect to HOQ. However these simplifications have a price: the whole process is blind toward competitors. Given these considerations, the use of the SHOQ approach can involve the risk of deploying more quality than needed thus increasing costs. Nevertheless we support the decision of the EDS SC management to use the SHOQ model: in its target market it is very difficult to collect detailed opinion about the competitors quality level. On the other hand, qualitative goals for a single project can be determined in a strict relation with the customer; hence the result of Step 2 in Table 2 can be a very precise one even in the absence of detailed information about competitors. Furthermore the risk is mitigated as EDS has, at the corporate level, functions monitoring competitor behavior; thus the solution centre does not have to devote resources to activities already performed elsewhere in the company. Finally, for large insurance or financial institutions the cost may or may not be the most relevant factor, whereas, it is for the downtime of a mission critical software. Although simplified, the SHOQ is well suited for the purposes of EDS SC; in fact, it allows the organization to obtain a clear perception of project quality goals. Moreover, SHOQ use is not limited to identifying quality goals: it permits a prioritization of them, which is a key point of CMM Level 4. These goals can be easily decomposed in measurable aspects of the features, hence controlled by SPC. It is worth noting that several practices already in place for the ISO compliance are used in Steps 1 and 2 in Table 2. Therefore, information already collected by the organization need only be integrated. This makes the model use easier than it seems at a first glance. Furthermore, the already in place PDSP, part of CMM Level 3, contributes to ease Step 3. The description of SQFD activities given in this subsection is forcedly linear in a waterfall fashion. As a final remark we note that in real processes it is frequent that quality goals, as is the case with other goals in software processes, have to be redefined during the project lifetime. Sometimes software quality goals conflict and actions must be taken to resolve these conflicts. The costs for achieving the project quality goals are analyzed and alternative quality goals are considered. Customers participate in quality tradeoff decisions. Software work products and plans are hence revised reflecting the tradeoff results.

17 3.4. Quantitative Process Management KPA In the Quantitative Process Management KPA, EDS SC applied its modified version of GDMP to define the set of metrics to collect and control. The ten steps of GDMP are shown in Figure 5 and compared in Table 3 with the nine steps of the EDS SC adapted process. 1 Business Goals Mental models To do this I will need to What do I want to achieve? 3 What do I want to know? Subgoals 2 <The Process> receives consists of produces holds entities entities entities 4 attributes attributes attributes Measurement Goals 5 G1 G2 Questions Indicators Measures Definitions Q1 Q2 Q3 I1 I2 I3 I4 M1 M2 M3 Definition checklist.. v.. v.... v v supplementa rules form Xx.. Xx.. Xx.. 9 Analysis & Diagnosis 10 Implementation Plan - Goals - Scope - Activities Figure 5. Goal Driven Measurement Process From Table 3 it is evident that there is a non complete correspondence between the two columns: EDS SC modified GDMP adding some steps and eliminating or modifying others. In particular EDS SC skips Step 4, groups Steps 6 and 7, groups Steps 9 and 10, and adds Steps 11 and 12. In the following we compare step by step the two methodologies indicating the rational of the mapping and the corresponding modified GDMP. Steps 1 to 3 are the same in both processes. The aims are: to define the set of business goals with the higher priority (Step 1); to investigate, understand, assess, predict, and improve the activities to achieve the goals (Step 2). Finally Step 3 translates, if necessary, complex goals into subgoals which are more specifically related to the activities. In Step 4 of GDMP, the mental models, and the entities and attributes associated with them are evolved using what emerged in Steps 1 to 3. Consequently Step 5 uses the updated mental models to formalize measurement goals. Within Step 5 of GDMP the first level of GQ(I)M is reached: goals are established.

18 Step GDMP EDS adapted process 1 Identify your business goals Identify your business goals 2 Identify what you want to know or learn Identify what you want to know or learn G 3 Identify your subgoals Identify your subgoals 4 Identify the entities and attributes related to your subgoals 5 Formalize your measurement goals Formalize your measurement goals 6 Identify quantifiable questions and the related indicators that you will use to help you achieve your measurements Identify indicators that you will use to help you achieve your measurement goals Q(I) goals 7 Identify the data elements that you will collect to construct the indicators that help answer your question M 8 Define the measures to be used, and make these definitions operational Define the measures to be used, and make these definitions operational 9 Identify the actions that you will take to implement the measures 10 Prepare a plan for implementing the measures Prepare a plan: your procedure for quantitative management 11 Metrics collection 12 Analysis and diagnosis Table 3. Goal-Driven Measurement Process vs. EDS adapted process EDS SC groups Steps 4 and 5 into Step 5. This means that the mental model is assumed available and does not need to be built ex novo each time. This assumption is justified by the fact that the model involved in establishing project goals in EDS SC is the previously presented version of QFD. This model allows the project leader to quickly determine what goals must be pursued in each project thus facilitating the passage from Step 3 to 5. The next two steps of GDMP, corresponding to the Question (Indicator) level of GQ(I)M, aim at defining quantifiable questions and formulating indicators to answer them (Step 6). In Step 7 these indicators are defined in terms of primitive metrics. EDS SC skips the explicit formulation of questions in Step 6 and directly ties indicators with business goals. Goals are very well detailed in subgoals, thus a direct relation to metrics is easily drawn. Moreover, EDS SC, similar to many other large organizations, has a standardized metrics plan common to all branches. These metrics are adopted in all projects. Unless there are specific customer requirements enforcing the collection of additional information the corporate defined software metrics are

19 collected. Indicators and goals are defined in the organization s document, the OSSP, whose template is reported in Figure 6. Using the OSSP, project leaders can establish which indicators are appropriate for a project and which metrics are involved. Step 8, the Measure level of GQ(I)M, defines the measures and how to collect them. In the EDS SC methodology this step is left unchanged. It is worth noting that it is also facilitated by the mentioned metrics plan. Step 9 and 10 of GDMP aim at making operational the collection of the selected metrics. EDS SC has a consolidated experience in metrics collections stemming both from CMM Level 3 practices and from the organization s attitude toward project activities measurement. Therefore the EDS SC methodology skips Step 9 leaving the project managers with the responsibility of deciding how to collect measures following company standard processes and practices. EDS SC adds to GDMP two other steps concerned with metrics collection (Step 11), and project measure analysis and diagnosis (Step 12). These steps make the whole methodology operative. During metrics collection the process control limits and the baseline are also defined. Figure 6. Organization Standard Software Process As briefly summarized, the methodology built by EDS SC to pursue the Quantitative Process Management KPA is largely based on GDMP. Nevertheless important changes have been made. These changes extend GDMP with operative steps while getting rid of the steps that can be avoided using other already established practices. In particular EDS SC can leverage practices that stem from CMM Level 3, such as the metrics plan, and from the corporate culture and body of practices (OSSP) to facilitate the model implementation. The result is a lightweight methodology compared to the original version; it maintains a clear separation of the involved steps and allows the organization to efficiently address the desired issues. Nevertheless some remarks must be made.

20 The process model built by EDS SC relies on some assumptions that, although realistic in EDS SC projects, must be outlined and checked to assess the overall methodology adequacy, and to gain generality. We will briefly sketch here the fundamental assumptions and we will discuss them later in further sections. The first assumption is that the goals are related to software maintenance activity; thus, despite customers and project specificities, the goals identified in each project are often the same. They are well known and previously analyzed at such a level of detail thus making the passage from subgoals identification (Step 3) and measurements goals (Step 5) a fairly simple one. The second important assumption deals with process stability: omitting Step 7 means it is possible to identify measures directly related to goals without an elaboration of the specificity of the project. Since EDS SC has a well defined process structure, very detailed company guidelines, and a metric plan built for CMM Level 3 compliance, these points are facilitated by already in place tools and methodologies. This may or may not be the case for other organizations. Thus it is in general essential to monitor and keep under control processes and goals stability to be confident with this methodology adaptation. 4. Organizational impact of CMM Level 4 Previous sections provide a somewhat static view of the implementation of the selected methodologies. We will now further detail how these methodologies have been implemented, how culture and organization changed, and the impact of the activities performed. In this section we first outline the essential steps carried out to prepare the organization for the CMM level 4 assessment, then attention is devoted to software metrics. Finally, we address the faced cultural and organizational changes and challenges which emerged during the whole process The process To prepare the organization for the Level 4 assessment, EDS SC followed a process composed of three macro phases: 1. Preliminary study; 2. Partial implementation; and 3. Organization wide implementation.

21 Early in the preliminary study phase the composition, the roles and the responsibilities of the team in charge of implementing the Level 4 practices were defined. The preliminary study required about two months. This phase enabled the organization to understand and assess the changes required by each KPA, choose the reference methodologies, and tailor them to its structure and needs. Deeply understanding the cultural gaps to overcome for adopting those methodologies turned out to be a very relevant issue. This is likely to be common to many software maintenance organizations: substantial training and a change in people s attitude are needed to go from CMM Level 3 to Level 4. Level 4 practices were subsequently implemented (i.e., experimentally applied in four pilot projects). This partial implementation phase was carried out in a pro-active way: programmers and project leaders actively cooperated in the identification of strengths and pitfalls. Most noticeably, this phase was performed in parallel with a training plan organized in collaboration with the University of Sannio to help project team members with the new approaches and tools, and more generally to bridge the cultural gaps. The last phase, ongoing at the moment of writing, addresses the improvement of the methodologies based on the results obtained in phase 2. It also focuses on the deployment of the road map to facilitate the application of the methodologies, and fosters the diffusion of the Level 4 perspective to the whole organization. As shown in Figure 7, the organizational structure set up by EDS SC to implement the improvement activities is composed of four layers. Level 4 Management Board SEPG SQAG Delivery Management Project team Project team Project team Figure 7. The Process Improvement Structure

22 The Level 4 Management Board includes the organization senior manager responsible for the whole SPI program plus other senior managers. Four software process specialists, trained in process improvement, compose the SEPG. They coordinate and control each process improvement activity. The Software Quality Assurance Group, SQAG, is responsible for project quality assurance: it reviews project activities and software products to guarantee adherence of the software project to the plans, procedures and standards. Both SEPG and SQAG report to the Level 4 Management Board. The Project teams are the teams working on pilot projects; several analysts and developers are coordinated by a project leader; the team size varies according to the project from a few units to up to a dozen programmers. The four pilot projects were selected according to requirements of stability and duration. All of them have an estimated completion date of about eighteen months; they have a fairly stable team composition and size. Teams were defined with a strict adherence to the company guideline. This is likely to enhance the comparability among these and among other projects in the company The metrics Collecting and analyzing software metrics is the cornerstone of Level 4 KPAs. Data collection goes together with the need for a corporate level database; this must be complemented by precise guidelines ensuring a common understanding of who, what, when and how to collect information, and how the collected information must be manipulated, encoded, and stored. In (Jalote, 2002), Jalote presents a study involving eight high maturity organizations; the study deals with software metrics collected by the different organizations and their use. Since the CMM does not specify which software metrics should be collected and how they should be used, one of Jalote s work goals was to discover similarities among organizations. The study shows that most of the organizations collect similar metrics and use similar metrics infrastructure (i.e., processes and database structure). Collected software metrics pertain to five areas: size, effort, schedule, defect, and risk. The metrics collected and used in EDS SC are those mentioned above plus metrics on project staffing, duration of activities, and changes. Collected primitive metrics are then used to derive composite measures and indicators. The SEPG team is responsible and manages the Organization Software Process Database (OSPDB); OS- PDB stores the historical data regarding closed projects as well as the data of ongoing activities. A template is provided to collect the values of all the considered primitive metrics over the life of the

23 projects; each team reports on a monthly basis the filled template to the SEPG. The collected values are compared (weekly or monthly according to the granularity of the monitored phenomenon) with organizational control limits Cultural changes As a CMM Level 3 organization has already made a considerable effort in order to reach and maintain CMM Level 3, it could be considered in some way prepared to face the next step. However, as highlighted in Section 2, the differences between Level 3 and Level 4 are substantial. First and foremost, moving up to Level 4 requires a considerable higher effort with respect to Level 3: Level 4 requires changing the way in which people perceive their relations with the organization and their work. Up to Level 3, software maintenance in a large organization may be conceived much more as a clerical job, with never changing processes, practices and a fairly high level of bureaucracy. Level 4 requires people to shift to a pro-active and more customer oriented approach, for example to act upon variables (e.g., changing the on going process) to stay within expected limits. Important differences between the two levels of the CMM that have an impact on the culture of the people involved are: a) Level 3 is focused on goals that can be obtained acting only on the variables that are inside the organization; on the other hand Level 4 involves the customer in the goal definition activity. Furthermore, the organization must be able to deal with variables that may or may not be controlled; b) Level 3 is aimed at implementing engineering practices and at allocating resources to pursue characteristics like high architectural maintainability. People with technical background are familiar with these goals. Level 4 is focused on the quality as perceived by the customers. A resource allocation taking into account what the customers think can be in contrast with the previous practices already in place; this can be distressful for people trained to think that their mission is simply to attain a technically sound solution. Moreover this dichotomy can lead to a sub optimal, hence uneconomical, resource allocation; and c) Level 4 imposes an active use of the collected metrics. Metrics must be used to make meaningful decisions, and to change resource allocations and people behaviors on the fly. The last point poses two major challenges. First, there is the need to acquire the technical skill to use SPC. The second aspect is even more relevant: there is the need of a consensus on the collected measures and on the use of the measures. At least, measures and indicators should be perceived as fair, correct, and