Baseline of Current Practise at ABB APR

Transcription

1 CODEN: LUTEDX ( TETS ) / 1-19 / (2001) & local 1 Baseline of Current Practise at ABB APR Thomas Olsson, thomas.olsson@telecom.lth.se Johan Nilsson, johan.l.nilsson@se.abb.com January 8,

2 Executive Summary A study to establish the current practise of the verification and validation process at ABB Automation Product AB (ABB APR) has been performed. The study is performed in cooperation between the Department of Communication Systems and ABB Automation Product AB (ABB APR). It is concluded that several basic process element are at place at ABB APR. The process used is basically a waterfall model. The process, called the Gate model, is largely followed. However, the process lack details and quality. The phases are defined, but the activities in each phase are ill-defined, both including purpose and content.this leads to a poor internal delivery quality, leaving too many defects in the product when entering the latter phases of testing and development. Also, the data collection procedures lack in granularity. Mainly the time collection procedures lack details. Finally, the overall training in verification and validation techniques is very low amongst the developers and testers. Baseline of Current Practise at ABB APR January 8,

3 1 Introduction ABB APR 1 and LUCAS 2 are engaged in a three-year project to improve the verification and validation (V&V) process at ABB APR. As an initial study, a baseline of current practise is established. The purpose of this study is to document current practise as well as identify potential weaknesses in the data collection procedures, which might make evaluation of process changes difficult. Also, the baseline is used as the starting point for the introduction of new elements in the V&V process. The purpose of this document is to present the current practise both to ABB APR personnel, as well as to LUCAS members. Outline of the report: Section 2 present the plan for the study and the data collection procedures. In section 3 the collected data is analyzed some improvement suggestions presented. The study as such is evaluated in section 4, for the purpose of future studies. Summary and conclusions are found in section Terminology GQM - Goal/Question/Metric paradigm. GQM is an approach to implementing measurement programs for improvement initiatives. See appendix A or [van Solingen 99]. V&V - Verification and Validation. Verification and validation is the collected procedures used to ensure product quality. See for example [Sommerville 95]. CM - Configuration Management. CM is the procedures explaining how configurations are defined. See for example [Asklund 99]. 2 Goal/Question/Metric Plan This section describes the plan for the study. The study is performed with a Goal/Question/Metric (GQM) approach. An introduction to GQM is found in appendix A. 2.1 Goal The goal is defined as suggested in [van Solingen 99]. Analyze the verification and validation process for the purpose of characterization with respect to effectiveness from the viewpoint of developers, testers and researchers in the context of the project Atlas 1. ABB Automation Products AB 2. Lund University Center of Applied Software Research Baseline of Current Practise at ABB APR January 8,

4 The goal takes a broad perspective, as there is limited data available on the processes in general. It is also assumed that the result will generate suggestions on needed improvements regarding metric collection. 2.2 Questions The questions are divided into three categories: Process conformance, Process domain understanding and Quality focus. Process conformance deals with issues regarding what process is being used and the procedures used in the process. For example, is a water fall model used? Process domain understanding is about the understanding of the process and the software domain. Issues considered are training, experience etc. The last category, Quality focus, involves questions regarding different quality aspects of the development. Examples are time spent and distribution of defects. The questions are summarized in table 1. As seen by the questions, a broad perspective is used. Note especially the emphasis on requirements. It is widely accepted that the quality of the requirements and the requirements process are important for efficient development of quality software. 2.3 Data Collection The data collection has taken four different forms: Interviews - Several of the questions are of a subjective or individual nature. These questions are answered through interviews. The interviews were conducted during one day by the authors. Eight persons were interviewed, varying from developers to testers and managers. Descriptive - Certain data is of a descriptive, unambiguous nature. These are extracted by consulting key personnel at ABB APR. For example, how is ABB APR organized? Quantitative - Data concerning time consumption and defect distribution are collected through quantitative measures from existing reporting facilities at ABB APR. Experience - By observing and studying at the workplace, certain aspects of the procedures are documented. As the terminology and background differs, this method is used to describe aspects in a way more available to the researcher than at ABB APR. The data collection took place during June and July, It should be noted that all collected information is treated anonymously. An elaboration on the metrics is found in appendix B. 3 Analysis of Data This section presents the baseline derived in the study. The analysis is divided into three parts: 1) Organization and Personnel, 2) Process and 3) Metrics. Organization and Personnel characterizes the organization at ABB APR and the background and training of the personnel. The analysis on the process describes the current practise at ABB APR, con- Baseline of Current Practise at ABB APR January 8,

5 TABLE 1. Question definition Process Conformance 1. Which test methods are used by the developers and testers? 2. Which inspection methods are used? Which documents are inspected? 3. Which group of people are responsible for producing the following test specifications: Unit test cases Integration test cases System test cases Acceptance test cases 4. Which group of people are responsible for performing the following activities: Document reviews Unit test Integration test System test Acceptance test 5. Which design methods are used? 6. What is the relationship between the following documents: Requirements Design Test cases 7. What kind of CASE tools are used, if any? 8. How is the company organized? Process definition Process Domain Understanding 9. How well do the testers and developers understand the requirements? 10.How well do the testers and developers understand the overall function? 11.How much experience do the testers and developers have with the product? Within the domain? 12.How is the general atmosphere towards changes and improvements to the development and testing process? Quality focus 13.What is the quality of the requirements? 14.How much time is spent in each phase by the different people involved in the project? 15.What is the lead-time of the development phases? 16.How does the distribution of faults look like over the different phases? 17.How many failures (defects still in the product) is observed in the delivered product? 18.How is the internal delivery quality perceived? 19.How many people are involved in the following activities: Requirement specification Development Testing One person might very well be involved in several activities 20.When are the different documents produced? cerning the development process in general and the V&V process specifically. In the metrics section analysis of data from the current project, Atlas, is presented. Organization and Personnel deals with issues regarding organizational structure, experience and training of personnel, etc. Process present the results from process related issues, such as phases, methods, measurements, etc. Metrics, the last category, present some conclusions from the time and defect reporting systems. Within the text at various places, certain improvement suggestions are presented. These should be seen as grains of ideas on how the development process can be improved. However, they are not further analyzed or described in detail. Any process changes based on this document should be evaluated separately. Baseline of Current Practise at ABB APR January 8,

6 3.1 Organization and Personnel Project and departments ABB APR in Malmö is organized according to the matrix organization principle. There are six departments, five developing departments and one product department. The Atlas project is staffed from the five development departments. Virtually all personnel in the different departments are involved in Atlas. Atlas consist of 12 sub projects. The development is done in parallel between the 12 projects. Apart from Atlas, there are two product projects, Victor and Eddie, staffed form the product department and from a separate department in Västerås respectively. The BCRAT (Big Configuration Release Acceptance Test) is performed in Västerås. There is also a project developing hardware, called Scorpio, staffed from a separate department Training and experience Most people working in the developing organization have a master s degree or equivalent, though not always in computer science or corresponding areas. Nobody of the interviewed people reported that they had any formal training in verification and validation. The level of degree varied more among the people involved in testing. This is partly because much of the testing done in the testing department is aimed at being user centric. It is therefore not appropriate to have domain experts performing the tests. The experience of all people involved in Atlas varies. The managers have generally worked at ABB APR (formerly Satt Control and Alfa Laval Automation) for many years and have a good general knowledge of the product and organization. However, many of the developers and testers have little or no previous experience within the domain. As with many companies in the software intensive market, personnel is hard to come by and since the technology moves very rapidly, keeping the pace with the development is difficult. This results in a high rotation of personnel and a large group of inexperienced developers. Suggestions. It is always difficult to allocate time for training. One way of handling this is to include training in the project budget. By allocating time and resources from the project it is more likely that there will be opportunities for training, rather than if this is done from some other organization. If there is a dead-line and something is not budgeted for, it will relentlessly be the first thing to be removed. 3.2 Process The development process used is basically a water-fall model, called the Gate model. The model defines deliverables at different stages in the process, called gates. The development is divided into several smaller projects, working in parallel on the product. However, all the groups still are bound to the gates, as they are in common for all sub projects. Baseline of Current Practise at ABB APR January 8,

7 Each sub project defines several activities. These vary in size from that one developer can do several activities during the development to several developers working on the same activity. Each activity is developed more or less independent of the other activities. If there are dependencies between activities, then scheduling takes place to prevent some activity from having to wait from someone else to finish. The following documents are defined: Market Requirement Specification, MRS Product Requirement Specification, PRS Functional Specification, FS Functional Test Specification, FTS (Unit testing) Integration Test Specification, ITS (Various tests) Product Type Testing, PTT (System testing) Big Configuration Release Acceptance Testing, BCRAT (Acceptance testing) Figure 1 depicts the development process, based on the documents and their causal order. Market and management Development departments Test department Västerås department MRS PRS FS FS FTS FTS ITS PTT BCRAT FS FTS FIGURE 1. Development process The MRS and PRS are produced by the market department and the project management jointly. From the PRS, sub projects and activities are defined and the development is continued in the different sub projects. The FS, FTS and ITS are produced by the development departments, as well as the development and maintenance coding. The final tests, the PTT and BCRAT are done by the test department and an acceptance test department respectively. The dashed circles in figure 1 can also be used do describe the overall phases in the process. The phases are elaborated on below The MRS and PRS In the first phase, the documents MRS and PRS are produced. This is done by market people and key personnel from the development organization. Baseline of Current Practise at ABB APR January 8,

8 3.2.2 The FS and FTS From the PRS, smaller projects are defined. For each project, several FS are written, for each identified function. When the implementation is done, functional tests are defined and performed. The steps are: Write FS, implement function, define and perform functional tests. This is the way the work is done in practice most of the time. According to the development process, the FTS is supposed to be written before or at least parallel to the development, but this is often not done. The FS and FTS has to be approved through a review before the subsequent step is initiated. Code reviews are performed periodically on all the code, not just the newly developed. Even though the process is well-defined and baselines are clear, they are not always followed. It is mainly the distinction between implementation and test which tends to be less defined. One reason is that defect reports are submitted continuously and the same developers who develop new code have to do maintenance updates as well. Code reviews are done sporadically and with varying degree between different sub projects. The steps in the process are well-defined, but the entry and exit criteria are not. Due to the fact that little or no training is received on how to write a functional requirements specification or how a test specification should look, the quality of these document is varied. Even though the FS and FTS are reviewed, they depend largely on senior developers to find flaws rather than a clear methodology or criteria for approval. Also, the instructions about the content of the FS and FTS is poor, adding to the varied character and quality. Suggestions. The Atlas project is delayed very much. Estimating the development time is always difficult. Since new development is mixed together with maintenance, estimation is made even more difficult. Also, the focus is somewhat lost when the two are mixed together. One solution is to define a separate project for maintenance or even a separate department. The benefits are: 1) More focused development and maintenance. 2) More accurate estimations. This may cause problems concern the configuration management. Many of the fixes in maintenance must be included into the new development as well. Due to the lack of independent modules, this problem is even greater. But with a well defined configuration plan and more independent modules, this problem would decrease The ITS The integration test specification is written by the developers or senior personal. The purpose is to test various things. The identification of which tests to perform is done by senior developers and test personnel. The areas which have had a lot of defects in pervious releases are used to identify test cases for the coming release. Again, a clear purpose and criteria on content of the integration test specification is ill-defined. The tests defined in the ITSs range from low-level structural (white-box) testing to high-level user-centric testing. The ITS are reviewed as all other documents. Baseline of Current Practise at ABB APR January 8,

9 Suggestions. The purpose of the integration tests is not clearly defined. By clearly defining the focus of the integration tests a higher effectiveness may be possible. It is seldom very effective to do random ad-hoc tests. One way of handling both integration test issues and testing of known trouble areas is to divide the integration tests into two separate testing activities: Integration tests and regression tests. By separating the activities, each of them can focus in the important issues and use techniques relevant to the specific activity. Typically, for regression tests historic data is used to identify areas where testing should be focused. However, this is not effective with integration testing as the focus there is mainly to test the integration of new code into the existing system The PTT and BCRAT After the system has passed all ITS, the product goes through an internal release, called beta, delivered to the test department for product type test (PTT). They perform various tests, purely from a user perspective, with test ranging from simple interactions tests to big configuration stress tests. No means of statistical testing or quality measurements are used. Rather, based on experience of key test personnel, decisions are made on courses of action. The test cases are defined fairly abstract, as they are intended to be real user situations rather than detailed instructions. The tests are logged and all defects are reported using a formal defect reporting system. The tests are of sampling character. That is, the system is tested only partly. Again, the test is based on experience of key personnel, not historic data or usage profile etc. The main difference from ITS is that only user aspects are considered, never details in the implementation Other Process Issues Formal defect reporting is introduced in the later stages of the process. As there are several parallel projects, it is not defined exactly when this is done. Rather, based on experience and on the current situation in the development project, management decides when to introduce the formal defect reporting. When this is done all defects have to be formally reported. Suggestions. It is always a waste of time reporting defects concerning only the developer him/herself. However, it can be of great importance to have a more formal defect reporting process throughout the development. The reporting is important to track the quality and changes in the system. One way of handling the reporting is to formally report all defects concerning baselined objects. Objects in this context can be a document, a protocol or a piece of code. This does of course require a defined CM process, where the baselining of objects is suitably defined. Baseline of Current Practise at ABB APR January 8,

10 3.3 Metrics With the current practice it is very difficult to extract quality data from the time and defect reporting systems. The focus in this section is on the collection procedures rather than an analysis of the data Time reporting The current time reporting is virtually non-existent. Only one development group has used a detailed time reporting. The other groups have only reported the sum of hours, not divided into activities or phases. This is largely due to major organizational changes. Time reporting is important to be able to evaluate changes in the working process. Also, the use of historic data for estimations is important. By using historic data, better estimations can be made than if management only use their intuition and experience. There are several aspects that are interesting to monitor: Activity - The activities can be new development, testing/reviewing, handling defect reports, meetings, training, etc. There might be other categories. Usually, an Other category is needed. Object - It is also interesting to see which objects that are worked on and how much. Objects in this context can be documents as functional specifications, test specifications or even source code. For some objects, typically source code, it can be useful to include more granular information. Project - If several projects run in parallel or if some activities are not related to the current project, it might be useful to include information on the context. It is important to note that the information should be easy to collect. It should not take much time from the people in their daily work. Also, it is important to have reasonable level of granular when reporting. It is probably not useful to be more granular than to the half hour. Considering the amount of time spent in a project and the amount of people, more granular information will be lost. Another important aspect, not time reporting specific, is the extraction of data. In any reporting system, there must be features easily available for extracting interesting data. If not, the data will never be used. Data that is not used will eventually not be collected and the data collection program is doomed to fail Defect reporting The procedures for reporting defects include logs from reviews and tests as well as a defect reporting system for individual defects. The latter is only used later in the development process. The exact instance in time when formal defect reporting is introduced is determined by project management. Baseline of Current Practise at ABB APR January 8,

11 In total there were 3755 defects reported from some kind of review and 162 defects from function test protocols and integration test protocols at the time of the data collection. The distribution on groups and specific reviews and tests are show in figure Automation Function Communication GUI HW Configuration Language Platf orm Runtime All 0 FSR CR TSR ITSR FTT ITT FIGURE 2. Defect detection The figure shows the number of defects per group, divided by the number of people in each group. All groups follow a similar pattern with a declining defect density. The low number of defects found in test indicate two things: It is likely that defects found in the functional tests and to some extent integration tests, are fixed without it being noted on any protocol. The tests lack in efficiency and a lot of tests are performed during other activities, informally. These conclusions can be drawn as most people involved in the development agree on that the code that has passed functional and integration tests often lack in quality. Hence, a lot of defects are missed during the two test phases. It is difficult to make any statement about the efficiency of reviews and tests. The time recording does not contain sufficient details. It is also difficult to relate to the lead-time as a lot of tests are performed informally and parallel to development of code or defect removal. There can be made no statements on the relative efficiency of different verification and validation efforts from figure 2. There is no group which display a behavior significantly different from the others. For example, if one group had a very low detection ratio during reviews and a high ratio in test this would indicate that reviews have an impact of latter test efforts. However, no such statement can be made without more detailed data. Baseline of Current Practise at ABB APR January 8,

12 At the time of the study there were submitted more than 1500 defect reports. The majority of these are submitted in the later stages of the development, as formal defect reporting was not required in the start of the project. The reports include both defects on newly developed code as well as maintenance reports. All the different types of reports are to a large extent lacking in three areas: Object - The object is difficult to extract from the reporting system. The concept of objects here is the same as with the time reporting, previously presented. Context - What activity lead to the uncovering of the defect? Was it discovered during test? Was a review session the reason for the uncovering? Insertion - To be able to make an educated choice of where to introduce changes in the process, it is important to consider when defects are introduced and when they could be removed. By including information on when the defect is introduced it is made possible to analyze where to introduce changes. For example, if a defect is introduced already in the requirements specifications, this should be removed before time is spent trying to implement the defect. Again, if the data is not easy to collect and extract, it won t be used and be doomed to fail. To be able to evaluate future changes to the V&V process, the above described data is important to collect. Also, if more detailed data is collected this can be used for estimating the quality at an early stage, making predictions on release dates more reliable. Suggestions. There are several reasons for reporting time and defects. One reason is to provide the management with information to help estimation of deadlines and quality. Another is to focus the efforts on the parts of the product where it is the most needed. A third reason is to be able to evaluate process improvements. If measurements are not available concerning time spending and defect distribution, it is very difficult to make any quantifiable statements about the outcome of an improvement. If a process improvement cannot be evaluated on other grounds than on the subjective sensation of the participating personnel then there is no accurate way of determine if the improvement is a success or not. 4 Validation of the GQM Study The purpose of this section is to analyze the study as such to revise the plan for subsequent studies. A goal in the overall project is to conduct a GQM study a couple of times during a year, to identify and document to the current practise. As a result of this analysis, new measurements are suggested to be introduced as well as a revision of the plan. Note that general process improvements are not the aim of this section. Rather, to improve subsequent studies, various aspects of the process has to be refined. These improvements are put forward for the purpose of subsequent studies. There are several of the questions in table 1 which could not be answered for different reasons. Most notable is the lack of data concerning time and defect reporting. Baseline of Current Practise at ABB APR January 8,

13 Question 15 and 20 relate to the phases definition in time. However, the phases are very long and do also not completely conform to the process model used. This information is probably not important enough to justify the efforts of collecting it under the current conditions. As a result, it is judged that these questions are better left to future studies and should not be considered in subsequent studies in the near future. Question 14 concern the effort, i.e. person hours, spent in the project on different activities. This is very important information. Even though it can be troublesome to collect it, it is the authors opinion that it is necessary to introduce this measurement. As it has recently been removed, it should provide little or no problem to introduce it ones again. The defect reporting is lacking in many areas. As discussed above, several aspects could be introduced. Most important is to introduce object and activity. This information is to a large extent already present, but not in an easily extracted form. By formalizing these issues, collection and extraction can be made more efficient. However, insertion information should probably be left for future changes. The main reason is the general maturity of the process as ABB APR and the fact that this has been tried before and failed. Also, too many changes should not be tried at ones. A couple of issues were overlooked in the original questions, but information is collected anyway. There are no specific questions on the test and requirement process, or on the overall development process. This information is interesting and easily collected. It is natural to include these in subsequent studies. There is a question concerning the motivation to improve. This questions needs to be elaborated to give a clearer view. Also, no attempt has been made to evaluate previous changes. By analyzing the introduction of other changes, an indication on motivation is possible. Therefore, it can be wise to analyze previous process changes. Also, the information on motivation is of poor quality, as it is a difficult issue to answer. Subsequent studies should try to improve the interview material to facilitate a better view of motivation. 5 Summary and Conclusions This report presents a study at ABB APR. The study is performed to describe the current practice of the verification and validation (V&V) process. This study is the first step in a three-year project to improve the V&V process. As an initial study, the focus is wide to get a general idea of the V&V activities. The study is performed using the Goal/Question/ Metric paradigm. The development process at ABB APR is a waterfall model with parallel development, called the gate model. The gate model defines several artifacts and when these should be completed. The artifacts are not described in detail regarding content and structure. This is the main problem with many of the produced documents. The quality varies a lot as well as content. Few or no methods, for example use case requirement analysis or UML design, are used to derive artifacts. Baseline of Current Practise at ABB APR January 8,

14 The V&V process includes reviews and tests. The reviews capture a lot of defects at an early stage. The teams performing the reviews consist of people from the developing group as well as test personnel and senior developers. The problem with the reviews relate to the lack of guidelines regarding content and structure of the review objects. The main problem with the tests performed is that a lot of testing is done informally and the formal testing lacks in structure and quality. As no specific method is used for testing, a lot of defects escape detection and the general delivery quality after functional and integration test is poor. The metrics collection is poor. Time reporting is lacking in most areas. This effectively leads to that detailed statements about the efficiency of V&V activities difficult to make. This fact also makes it difficult to evaluate future changes, as little quantitative data is available. The defect reporting lacks in detail, but the main parts are present. The most important issue missing in the defect reporting is the relationship with document or code module. Unless this information is available, the quality of the product is difficult to estimate before release. Baseline of Current Practise at ABB APR January 8,

15 References Asklund 99 Sommerville 95 van Solingen 99 Wohlin 00 Asklund, U. Configuration Management for Distributed Development - Practice and Needs. Licenciate thesis, Department of Computer Science, Lund University, Sommerville, I. Software Engineering. Fifth Edition, Addison-Wesley, van Solingen, R., Bergout, E. The Goal/Question/Metric Method: A Practical Guide for Quality Improvement of Software Development. McGraw-Hill Publishing Company, UK Wohlin, C., Runeson, P., Höst, M., Ohlsson, M.C., Regnell, B., Wesslén, A. Experimentation in Software Engineering: An Introduction. Kluwer Academic Publishers, Boston, MA, USA Baseline of Current Practise at ABB APR January 8,

16 Appendix A Introduction to GQM The Goal/Question/Metric paradigm is designed to support improvement and data collection programs. A problem in many empirical studies is the lack of relevant data. Data may be collected without a clear purpose. The result is a waste of a lot of effort collecting data nobody has any use for. GQM tries to tackle the problem with useless data by prescribing a top-down approach. There are three basic steps: Goal establishment - The goal is the first thing to be established. This includes context, viewpoint, type of study etc. Question derivation - After the goal has been established, questions are derived to achieve it. There may be many sorts of questions, ranging from descriptive ones to questions involving absolute measurements. Metrics definition - The last step is the metrics definition. The metrics describe how the questions can be answered through various data collection. The data collection ranges from interviews to extracting information on defects from reporting systems. It is possible that a metric answers more than one question and that one question requires more than one metric to be answered. By first establishing the goal, it is more likely that the data collected will be more useful than if data is collected before the goal is defined. This is a widely used approach within the software engineering research community involved with empirical research. For further information on GQM, see for example [van Solingen 99]. Appendix B Metrics definition An explanation of categories of measurement type and object are found in table 2 and table 3 respectively. The descriptive measurements are typically collected by observation or by key personnel at ABB APR. The ordinal metrics are mostly collected through interviews with people involved in the development. Lastly, The absolute metrics are collected by extracting information from existing reporting system, both time data and data on defects. For more details on the metrics, see for example [Wohlin 00]. It should be noted that the categories are not entirely exclusive. Table 4, 5 and 6, respectively, describe the metrics, divided into the same categories as the question in table 1. TABLE 2. Metrics definition Descriptive name Descriptive (D) Explanation The answer is put in general, non-quantifiable terms. For example, boundary value analysis is used when constructing unit tests. Baseline of Current Practise at ABB APR January 8,

17 TABLE 2. Metrics definition Descriptive name Ordinal Scale (O) Absolute scale (A) Explanation The answers can be compared to each other, but have no absolute value. For example, understanding of requirements might be better or worse for two different persons. However, it is not possible to quantify the understanding. The answer is quantifiable and has a absolute meaning. For example, number of testers involved with system testing. TABLE 3. Measurement object Descriptive name Process (Proc) Explanation The answer concerns the development process as such. For example, which phases is used in the process. Resource (R) The answer is a description of how many resources is used. For example, amount of time. This metric also include people metrics. For example, the experience of a tester. Product (Prod) The answer is a description of the product or environment on which the survey is being conducted. For example, number of faults found in the product after delivery to customer. The metrics for process conformance in table 4 are all of descriptive nature. The aim is to describe the current process in general terms. TABLE 4. Metrics definition for process conformance Question number Metrics attribute Comment 1 D, Proc The specific techniques and test phases is sought. On phases, answer in terms of unit, integration, system, acceptance test and so on. On techniques answer in terms of black box, white box, boundary analysis, coverage, statistical testing, automatic testing, regression testing and so on. Also, indicate how test stopping criteria is defined. 2 D, Proc Which documents and which code is reviewed? Also state how the review groups are formed and how the reviews are being performed. 3 D, Proc The responsible for different documents is sought. Consider the following document: Market requirements, project/product requirements, functional requirements, unit test cases, integration test cases, system test cases and acceptance test cases. It is not individual names sought. Rather their organizational position, e.g. project manager. 4 D, Proc The people responsible for performing the different activities are sought. Indicate who performs: Document reviews (including code reviews), unit test, integration test, system test and acceptance test. 5 D, Proc The answers here consider design guidelines and methods. Is the design implicit or explicit? Are any design methods used? Required by the management? Baseline of Current Practise at ABB APR January 8,

18 TABLE 4. Metrics definition for process conformance Question number 6 D, Proc State the relationship between all of the following documents: Market requirement specification, project requirement specification, functional specification, unit test cases, integration test cases, system test cases, acceptance test cases and design documents. 7 D, Proc Working with the different activities, what kind of CASE tools is being used? Consider requirements, design, code and test. Is configuration management used? 8 D, Proc State how project groups are organized and the departmental organizational structure. Also, how are the different structures related. Table 5 depicts the process domain understanding. The information here is typically ordinal. That is, it is possible to relate the answer to other answers, but the answer is not meaningful by itself. TABLE 5. Metrics definition for process domain understanding Question number Metrics attribute Comment 9 O, R Ask a couple of people within the development project regarding requirements understanding. Interview questions are found below. 10 O, R As previous comment. 11 O, R Interview questions for a qualitative description of the experience of the people involved within the project, concerning process and domain experience. Interview questions are found below. 12 D, R Interview questions for a qualitative description of the motivation to change and develop as a person part of the organization. The metrics for the quality focus, found in table 6, contain more quantifiable metrics. TABLE 6. Metrics definition for the quality focus Question number Metrics attribute Metrics attribute Comment Comment 13 O, Prod This answer should indicate the quality of the requirements. Typical attributes are understandability, stability and level of detail. There exist other aspects which might be interesting, but for practical reasons limitations are put on the answers. 14 A, R Indicate how much time is spent on different activities at different phases. It is sought how many working hours is spent on different activities during the project. Phases are: pre-study, project specification, design, code, unit test, integration test and system test. 15 A, R Characterize the lead-time for different phases and, if applicable, phase overlap. 16 A, Prod Characterize the fault distribution on the product. Classify faults with respect to severity and type. Also indicate when the faults are injected, detected and removed, considering both lead-time and phase. Baseline of Current Practise at ABB APR January 8,

19 TABLE 6. Metrics definition for the quality focus Question number Metrics attribute Comment 17 A, Prod Indicate how many faults that were left in the delivered product and that caused a failure. Also, classify the defects with regard to severity and type. 18 O, Prod Characterize the quality of internally deliverable. Deliverable are: Market requirement specification, project requirement specification, functional specification, code (program) and the results of the different test activities. Interview questions are found below. 19 A, R State the number of people involved in different activities through out the project. A person might participate in many activities. Activities in this context are: Requirements specification (market and project), management, code development and unit test, integration test, system test and acceptance test. 20 O, Prod Do a pair-wise comparison between relevant documents to see their causal relationship. Baseline of Current Practise at ABB APR January 8,