Chalita Thongchua Wenjin Yang

Transcription

1 Master Thesis Software Engineering Thesis no: MSE March 2007 Correlations between Requirement Attributes and Process Attributes - Identifying and quantifying the correlations in a rapid software development process Chalita Thongchua Wenjin Yang School of Engineering Blekinge Institute of Technology Box 520 SE Ronneby Sweden

2 This thesis is submitted to the School of Engineering at Blekinge Institute of Technology in partial fulfillment of the requirements for the degree of Master of Science in Software Engineering. The thesis is equivalent to 40 weeks of full time studies. Contact Information: Author(s): Chalita Thongchua Wenjin Yang External advisors: Pierre Gustafsson Lars-Ola Damm Ericsson AB University advisor: Daniel Häggander School of Engineering School of Engineering Blekinge Institute of Technology Box 520 SE Ronneby Sweden Internet : Phone : Fax : ii

3 Abstract It is reported that the market-driven product development is becoming common in the software industry. There are two challenges in the market-driven product development: time-to-market and meeting customers requirements. A rapid software development process is regarded as a good way to solve those two challenges. Streamline development process (SLDP) is aligned with a rapid software development process, which is an in-house development process of Ericsson AB. In this study, seven completed projects from the streamline development process were investigated. The correlations between requirement attributes and process attributes were identified and quantified in the SLDP. Nine hypotheses were assumed. Four hypotheses were derived from the correlations from the other software development processes, and the other five hypotheses were derived from new requirement attributes and process attributes in SLDP. Two statistical software applications were used in the hypotheses testing. The results of those hypotheses showed that too much time spent in the early phase of streamline development would not reduce the time to market. A SLDP measurement program contains the measurements of requirement attributes and process attributes. This measurement program was mainly composed of four core attributes (size, effort, schedule, and fault), the requirement volatility, the completeness, the resource overrun, and the estimation accuracy. The results of the SLDP measurement program reflected four challenges in the SLDP: the requirement engineering process, the release planning, the estimation accuracy at each development phase, and the quality of the documentation. At last, based on those four challenges (the requirement engineering process, the release planning, the estimation accuracy at each development phase, and the quality of the documentation) and the defined correlations between requirement attributes and process attributes in the SLDP, the improvement opportunities were proposed for the SLDP. Keywords: a rapid software development process, requirement engineering, measurement, correlation

4 Acknowledgements First of all, we would like to thank Ericsson for giving us the probability to study their work. This thesis was accomplished with the help of some people who we would like to express our thankfulness. Thank Daniel Häggander for his patience and guide on criticizing the contents of this thesis, especially research method. We would also like to thank our Ericsson AB PAU/PAY advisors, Pierre Gustafsson and Lars-Ola Damm, for providing support and suggestions. All of them put much effort on reviewing and criticizing this thesis. We would also like to express our gratitude to project managers and their colleagues in Ericsson AB, who participated in questionnaire and interview sessions. Thanks to their help, we can solve problems during the investigation. Finally, thank our families and friends for their support during the time we studied at BTH.

5 Table of Contents LIST OF FIGURES LIST OF TABLES III IV 1 INTRODUCTION 1 2 A RAPID DEVELOPMENT PROCESS STREAMLINE DEVELOPMENT PROCESS CHARACTERISTICS OF SLDP 5 3 REQUIREMENTS IN A DEVELOPMENT PROCESS IMPORTANCE OF REQUIREMENTS THE CORRELATIONS REQUIREMENT HANDLING IN SLDP 7 4 SOFTWARE PROCESS MEASUREMENT FOUR KEY ATTRIBUTES REQUIREMENT ATTRIBUTES PROCESS ATTRIBUTES 9 5 RESEARCH METHOD HYPOTHESES ON THE CORRELATIONS A SLDP MEASUREMENT PROGRAM DATA COLLECTION DATA ANALYSIS VALIDATION Construct Validity Internal Validity External Validity Reliability 16 6 MEASUREMENT RESULTS MAIN REQUIREMENT LEAD TIME MAIN REQUIREMENT SIZE REQUIREMENT VOLATILITY PROJECT LEAD TIME The feasibility study phase The implementation and testing phase The latest system version phase The release phase PROJECT SIZE PROJECT FAULT DENSITY COMPLETENESS RESOURCE OVERRUN ESTIMATION ACCURACY The project initiated date The estimation accuracy of total resource The estimation accuracy of time between QD THE CHALLENGES IN SLDP 28 7 HYPOTHESIS RESULTS HYPOTHESIS ONE HYPOTHESIS TWO HYPOTHESIS THREE HYPOTHESIS FOUR HYPOTHESIS FIVE 33 i

6 7.6 HYPOTHESIS SIX HYPOTHESIS SEVEN HYPOTHESIS EIGHT HYPOTHESIS NINE THE CORRELATIONS 38 8 IMPROVEMENT OPPORTUNITIES 41 9 CONCLUSION FUTURE WORK REFERENCES 45 APPENDIX Ⅰ 48 APPENDIX Ⅱ 50 ii

7 LIST OF FIGURES Figure 1 The streamline development process [Tomaszewski et al. 06]...4 Figure 2 Relationship of requirements to other development processes [Wiegers 03]...6 Figure 3 A main requirement from start to finish...7 Figure 4 The main requirement lead time...17 Figure 5 The main requirement size...18 Figure 6 The average lead time and the standard deviation between the grouped QD...20 Figure 7 The average lead time and the standard deviation at the feasibility study phase...20 Figure 8 The reasons for the delay at QD2 assessment...21 Figure 9 An average lead time at the implementation and testing phase...21 Figure 10 The deviations at QD5 assessment...22 Figure 11 A project time to market...22 Figure 12 The standard deviation of the lead time between QD...23 Figure 13 The project size...24 Figure 14 The estimated and actual days at QD6-Released...28 iii

8 LIST OF TABLES Table 1 A guideline for the interpretation of a correlation coefficient [@Wikipedia]...15 Table 2 The requirement volatility...19 Table 3 The fault density...24 Table 4 The resource overrun...25 Table 5 The estimation accuracy of total resource...26 Table 6 The estimation accuracy of time between QD...26 Table 7 The data distribution of the lead time at QD0-QD2 and QD0-QD Table 8 The data distribution of the lead time at QD0-QD2 and the MRE on total schedule30 Table 9 The data distribution of the lead time at QD0-QD2 and the MRE on total cost...31 Table 10 The data distribution of the lead time at QD0-QD2 and the MRE on total effort..31 Table 11 The data distribution of the lead time at QD0-QD2 and the schedule overrun...32 Table 12 The data distribution of the lead time at QD0-QD2 and the project cost overrun..32 Table 13 The data distribution of the requirement volatility and the project cost overrun...32 Table 14 The data distribution of the requirement volatility and the project schedule overrun...33 Table 15 The data distribution of the grouped MRLT and the lead time at QD0-QD Table 16 The data distribution of the grouped MRLT and the lead time at QD0-QD Table 17 The data distribution of the grouped MRLT and the lead time at QD1-QD Table 18 The data distribution of the grouped MRLT and the lead time at QD2-QD Table 19 The data distribution of the grouped MRLT and the lead time at QD3-QD Table 20 The data distribution of the grouped MRLT and the lead time at QD4-QD Table 21 The data distribution of the grouped MRLT and the lead time at QD5-QD Table 22 The data distribution of the grouped MRLT and the lead time at QD6-Released..36 Table 23 The data distribution of the grouped MRLT and the MRE of total schedule...36 Table 24 The data distribution of the grouped MRLT and the MRE of total cost...36 Table 25 The data distribution of the grouped MRLT and the MRE of total effort...37 Table 26 The data distribution of the grouped MRLT and the schedule overrun...37 Table 27 The data distribution of the grouped MRLT and the cost overrun...37 Table 28 The data distribution of the grouped MRLT and the project effort overrun...38 Table 29 The data distribution of the lead time at QD0-6 and QD6-Released...38 Table 30 The correlations...39 iv

9 1 INTRODUCTION The market-driven product development is becoming common in the software industry [Butscher & Laker 00] [Gorschek & Wohlin 06] [Ruhe & Greer 03]. Compared with a customized software development, the market-driven product development is more urgent to deliver the products on time and meet customers requirements [Host et al. 01] [Sawyer 00]. A rapid software development process (such as agile methods, extreme programming) is regarded as a good way to reduce time to market and meet customers requirements, because a series of product version can be released to market and customers can propose new changes based on the released product versions [Sommerville 04]. Aligned with a rapid software development process, the streamline development process (SLDP) has been applied since 2005 at Ericsson AB. It contains five phases: the pre-study, the development project phase, the latest system version (LSV) phase, the maintenance phase, and the release project phase. Up to now, only one paper had discussed about the disadvantages and advantages of the SLDP, and it was an early evaluation of the streamline development process [Tomaszewski et al. 06]. Thus, a measurement program is required to assess and improve the current situation of SLDP at Ericsson AB. Both academic research and industrial evidences have already been proved that a requirement engineering process plays an important role in the software development life cycle [Dahlstedt & Persson 05] [Damian et al. 05] [Palyagar 04] [Regnell & Brinkkemper 05] [Wiegers 03] [Young 04]. The problem is that the percentage of resources devoted into requirement engineering varies a lot. Wiegers suggested that 12 or 15 percent of total effort is spent on requirement engineering [Wiegers 03]. Others also suggested that 15.7 or 25 percent of total effort is spent on requirement engineering [Hofmann & Lehner 01] [McPhee & Eberlein 02]. Under the circumstances, project managers have to plan how much effort should be devoted into requirement engineering, and decide the best time to start implementing and testing the projects in different development processes [Wiegers 03]. Therefore, it would be valuable to investigate the requirements and explore the correlations between requirement attributes and process attributes in the SLDP. The research question is listed below: Research Question: What are the correlations between requirement attributes and process attributes in the streamline development process? To solve the research question, the framework was designed. The correlations between requirement attributes and process attributes in the streamline development process were assumed. A measurement program was set to measure the requirement attributes and process attributes in the streamline development process. These measurements are a prerequisite for the hypothesis testing. The data from this measurement program would be used to prove the assumed correlations between requirement attributes and process attributes. The improvement opportunities were based on the result we have found in this study. Due to the time limitation and the data availability, nine main hypotheses were assumed in this study. Four hypotheses had been investigated in other software development processes. The other five hypotheses were assumed without any theoretical and industrial evidences. This main motivation was from the characteristics of the SLDP: the main requirement analysis at the pre-study phase and at the 1

10 feasibility study phase, the separation between the latest system version phase and the release phase. Hypothesis 1: The longer feasibility study phase can save the project development time [Wiegers 03]. Hypothesis 2: The longer feasibility study phase can lead to the better project estimation accuracy of schedule, cost, or effort. [Wiegers 03] Hypothesis 3: The longer feasibility study phase can decrease the probability of the project resource overrun [Hooks and Farry 01] Hypothesis 4: The lower requirement volatility can decrease the probability of project resource overrun [Zowghi & Nurmuliani 02] Hypothesis 5: The longer pre-study phase can save the project development time. Hypothesis 6: The longer pre-study phase can reduce the development project lead time at each phase. Hypothesis 7: The longer pre-study phase can lead to the better project estimation accuracy of schedule, cost, or effort. Hypothesis 8: The longer pre-study phase can decrease the probability of project resource overrun. Hypothesis 9: The longer project development time can prolong the project time to market. At first, a SLDP measurement program was composed of the measurements of requirement attributes and process attributes. Those measurements were the starting point of hypothesis testing. Based on the literature review, this measurement program was mainly composed of four core attributes (size, effort, schedule, and fault), the requirement volatility, the completeness, the resource overrun, and the estimation accuracy. A document inspection was used as a main way in the data collection. To complement the incomplete data and investigate the reasons, a questionnaire and some specific s were sent to project managers, and an interview was also conducted. During this investigation, we found four challenges in the SLDP, which are related to the requirement engineering process, the accuracy of the release planning, the estimation accuracy at each development phase, and the quality of documentation. Two statistical software applications (STATGRAPHICS Centurion XV version and StatsDirect version 2.6.2) were used in the hypotheses testing. We decided to accept the hypotheses, whose p-value is lower than 0.5. It means that there would be higher than 50% of probability to get the same conclusion. The results show that the long pre-study phase or the longer feasibility study phase would not definitely reduce the product time to market. At last, based on those four challenges and the defined correlations in the SLDP, the improvement opportunities for the SLDP were suggested: reducing the main requirement lead time, utilizing the requirement management tools properly, improving the accuracy of the release planning, allocating the appropriate resources at each development phase, and improving the quality of documentation. 2

11 The outline of the remaining chapters is as followed. In Chapter 2, an overview of SLDP and the characteristics of SLDP are introduced. In Chapter 3, the importance of a requirement in a software development process and the correlations between requirement attributes and process attributes are presented. Moreover, the requirement handling process in SLDP is followed. In Chapter 4, the measurements of the requirement attributes and the process attributes are presented. In Chapter 5, the research method is presented and validated. In Chapter 6, the results of the SLDP measurement program are analyzed. In Chapter 7, the correlations between requirement attributes and process attributes in the SLDP are investigated. In Chapter 8, the improvement opportunities for the SLDP are presented. At last, the conclusion and future work of this study is presented in Chapter 9 and Chapter 10. 3

12 2 A RAPID DEVELOPMENT PROCESS Rapid Application Development is any development methodology or activity whose overall impetus is to increase speed of application development over that of the traditional development life cycles. [McPhee & Eberlein 02] The difference between the traditional development processes and the rapid development processes is that the rapid development process can deliver increments with new functions at regular intervals [Sommerville 04]. The purpose of this chapter is to introduce the StreamLine Development Process (SLDP). 2.1 StreamLine Development Process Aligned with a rapid software development process, the streamline development process (SLDP) has been applied since 2005 at Ericsson AB. As it is shown in Figure 1, it contains five phases: the pre-study phase, the development project phase, the latest system version (LSV) phase, the maintenance phase, and the release phase. Opportunity (customer, innovative, market-trend) Requirement Repository Pre-Study Development project LSV Release LSV Requirement Package Development Project A Release Requirement Package Development Project B Requirement Package Design maintenance Potential Release Development Project C Requirement Package Development Project D Time Figure 1 The streamline development process [Tomaszewski et al. 06] - The pre-study phase At first, the product requirement board would handle and prioritize the requirements from the business view. Then, some main requirements would be rejected, and some main requirements would be approved. Those approved main requirements would be analyzed from the technical perspective to estimate technical risks and the development capacity. 4

13 - The development project phase Based on the technical analysis in the pre-study phase and the restriction of three months development time, the program planning board would divide those approved main requirements into several development projects. Each development project is composed of five phases: the project planning phase, the designing phase, the implementation phase, the functional testing phase, and the non-functional testing phase. - The latest system version (LSV) phase Each completed development project is integrated with other development projects in this phase. Then, the latest system version is produced. This completed latest system version is called a potential release version. - The release phase In this phase, the potential release version is selected and released to market. - The design maintenance phase The design maintenance teams handle the trouble report (TR) from the LSV phase and end customers. If the correction package contains a new feature of the product, it is integrated with the latest system version. 2.2 Characteristics of SLDP In sum, three main characteristics of this software development process are listed below: - The main requirement analysis at the pre-study and the feasibility study phase At first, each main requirement is analyzed at the pre-study phase. The most prioritized main requirements are selected and allocated to a development project team at the prestudy phase. Second, when a development project is initiated, those main requirements are investigated at the project feasibility study phase (the project planning and designing phase). - The development project scope Each development project scope is most likely to be smaller than other software development projects, since the development time is usually not more than three months. [Tomaszewski et al. 06] - The latest system version phase and the release phase Not all the latest system versions are going to be released to market, which depends on the release planning. Therefore, there is a partition between the latest system version phase and the release phase. [Tomaszewski et al. 06] As we mentioned before, the SLDP has been applied for one year. Only one paper [Tomaszewski et al. 06] discussed about the feasibility of applying the SLDP at Ericsson AB. Thus, a measurement program is required to assess and improve the current SLDP at Ericsson AB. 5

14 3 REQUIREMENTS IN A DEVELOPMENT PROCESS In this chapter, the importance of a requirement in a software development process is explained. Then, the correlations between requirement attributes and process attributes are introduced. At last, the requirement handling process in SLDP is presented. 3.1 Importance of Requirements It has been acknowledged that the requirement engineering process plays an important role in a software development process [Aurum & Wohlin 05] [Damian et al. 05] [Dahlstedt & Persson 05] [Palyagar 04] [Regnell & Brinkkemper 05] [Wiegers 03] [Young 04]. Figure 2 shows the relationships of requirements to other development processes: the project planning process, the design and coding process, and the testing process [Wiegers 03]. Baselined Requirements Project Plans Designs & Code - Use requirements to size the project - Base estimates on product size - Update plans as requirements change -Use requirement priorities to drive iterations - Have developers review requirements - Use quality attributes to drive the architecture - Allocate requirements to components - Trace requirements to designs and code Figure 2 Relationship of requirements to other development processes [Wiegers 03] - The project planning process The planners can use the textual requirements (requirement specification) to estimate the product size and the required resources. The project plans have to be updated as a requirement changes. Requirement priorities are also used to drive iterations. [Wiegers 03] - The design & coding process Developers review the requirements to design the product. Then, the requirements would be allocated to the components. [Wiegers 03] - The testing process The functional and non-functional testing is based on the requirements. Then, the requirements can be traced. [Wiegers 03] 3.2 The Correlations Tests - Start test design early - Use requirements to drive system testing - Have users develop acceptance tests - Trace requirements to tests As we mentioned before, some researches have proved that the requirement engineering process can affect the software development process. The following is the correlations on how the requirement engineering process can affect the software development process: 6

15 Correlation 1: The appropriate effort devoted in the requirement engineering can save the development time [Wiegers 03] Correlation 2: The more effort devoted in the requirement engineering can lead to the better estimation accuracy of resource. [Wiegers 03] Correlation 3: The more effort devoted in the requirement engineering can decrease the probability of the resource overrun [Hooks and Farry 01] Correlation 4: The lower requirement volatility can decrease the probability of cost overrun and schedule overrun [Zowghi & Nurmuliani 02] When it comes to the percentage of effort that should be devoted into the requirement engineering, the opinions are different. Wiegers suggested that we can spend 12 or 15 percent of total effort in the requirement engineering [Wiegers 03]. One empirical study shows that 15.7 percent of total effort and 38.6 percent of total schedule should be devoted into the requirement engineering [Homfmann & Lehner 01]. Hooks and Farry found that 10 percent of resources in the requirement engineering would decrease the probability of cost overrun and schedule overruns [Hooks & Farry 01]. Moreover, another survey is reported that 25 percent of total effort should be allocated in the requirement engineering [McPhee & Eberlein 02]. Under the circumstances, project managers have to plan how much effort should be devoted into the requirement engineering, and decide the best time to start implementing and testing the projects [Wiegers 03]. 3.3 Requirement Handling in SLDP In general, a market-driven requirement engineering process is composed of the requirement elicitation, the requirement analysis, the requirement prioritization, the requirement selection, the requirements validation, and the requirements management [Gorschek 06]. Figure 3 shows how a requirement is through the SLDP at Ericsson AB. QD-A QD-B QD0, QD1, QD2, QD3, QD4, QD5 QD6 Requirement Definition System Analysis Development LSV Opportunity Approval Release Preparation Release Handling GA Maintenance ICP, EP Figure 3 A main requirement from start to finish As it is shown in Figure 3, a main requirement is assessed by the quality door (QD) in the SLDP. The following list is the main goal of each QD assessment: - QD-A: the main requirements are approved - QD-B: the business decision is made and the system analysis is initiated - QD0: the development project scope is defined and the feasibility study is started. - QD1: the requirement definition is completed 7

16 - QD2: the implementation and the test scope is defined - QD3: the implementation and the integration test is completed - QD4: the functional testing is completed - QD5: the non-functional testing is completed - QD6: the latest system version is completed. 8

17 4 SOFTWARE PROCESS MEASUREMENT This chapter would introduce the measurements of the requirement attributes and the process attributes. 4.1 Four key attributes There is an amount of measurable attributes in a software development process [Florac et al. 97] [McGarry & Jones 94] [Zuse 98]. Among those attributes, four key attributes can be easily measured through a software development process [McGarry & Jones 94]. Those four key attributes are: size, development effort, development schedule, and software error [McGarry & Jones 94]. Due to the time limitation and data availability, those four key attributes are chosen in this study. 4.2 Requirement attributes Amount of measurements have also been defined on the quality of the requirement specification, a requirement development process, and a requirement management process [Costello & Liu 95] [Loconsole 01] [Mora & Denger 03] [Wang & Lai 01] [Yilmaztürk 05]. However, we would only introduce the measurements which are applicable in the SLDP. - requirement lead time It is the time from the start date to the ending date in the requirement repository. - requirement size It can be measured by the number of test cases for each main requirement [Wiegers 03] - requirement volatility The number of the requirement change requests and the total number of main requirements are recorded. And the requirement volatility can be computed to be: added + modified+ deleted req. Requirement volatility = *100% [Wiegers 06] initial number of req. 4.3 Process attributes The following list is the measurements related to the process attributes. - project lead time It is the time spent at the different development phases [Florac et al. 97]. - project size It can be measured by the actual total man-hour of project. This measurement was also confirmed by the project managers in the questionnaire. 9

18 - project fault density The project fault density could be computed to be: number. of. fault fault. density = *100% [Mohagheghi et al. 04] project. size - completeness The assessment of the presence and agreement of all necessary software system parts. [IEEE 982.1] It can be computed to be: # output. requirement completene ss = *100% [Salamon & Wallace 94] # input. requirement - resource overrun (effort, cost, and schedule) We would compare the actual values with the estimated values of total effort, cost and schedule. If it is resource overrun, this attribute is assigned to be 1. Otherwise, this attribute is assigned to be 0. [Zowghi & Nurmuliani 02] - estimation accuracy (effort, cost, and schedule) To measure the estimation accuracy, there are five steps [Fenton & Pfleeger 98]: 1. The relative error (RE) is computed to be: Actual Estimate RE = (If the relative error is negative, then it is Actual overestimated. If the relative error is positive, then it is underestimated.) 2. The magnitude of the relative error (MRE) is computed to be: Actual Estimate MRE = (i.e. MRE is the absolute value of the relative error) Actual 3. The mean relative error for n projects is computed to be: n 1 RE = RE i n i= 1 4. The mean magnitude of relative error is computed to be: n 1 MRE = MRE i n i= 1 5. The estimation accuracy is computed to be: PRED ( q) = k / n (where q is assigned to be 0.25, k is the number of item whose magnitude of the relative error is less than q, and n is the number of projects). Usually, the estimation technique is acceptable if PRED(0.25) is at least

19 5 RESEARCH METHOD As we mentioned before, this streamline development process has been applied since the late 2005 at Ericsson AB. This is a rather new development process. A measurement program is required to understand the current situations in the streamline development process. Followed with the results of the measurement program, the improvement opportunities could be brought about. Our main research issue is to explore the correlations between requirement attributes and process attributes in the streamline development process. The correlations could also be used in the improvement opportunities part. The research question is listed below: Research Question: What are the correlations between requirement attributes and process attributes in the streamline development process? To solve the research question, nine main correlations between the requirement attribute and the process attribute were assumed. Then, a measurement program was designed to define the measurements of the requirement attributes and process attributes. Those measurements are the starting point of the hypothesis testing part. The data from the measurement program is used to calculate the correlation coefficient. Based on the result of the measurement program and the correlations, the challenges were summarized and the improvement opportunities in SLDP were suggested. The multiple cases study was selected as a research strategy [Yin 03]. There are seven projects, and they are all completed projects developed in the streamline development process. They were from the same product line. A document inspection was used as a main way to collect data. A questionnaire, an interview and some specific s were also used to complement some missing data and investigate the reasons. 5.1 Hypotheses on the Correlations Due to the time limitation and the data availability, nine hypotheses were assumed. Four hypotheses had been investigated in other development processes. The other five hypotheses were assumed without any theoretical and industrial evidences. This main motivation was from the characteristics of the SLDP: the main requirement analysis at the pre-study phase and at the feasibility study phase, the separation between the latest system version phase and the release phase. Hypothesis 1: The longer feasibility study phase can save the project development time [Wiegers 03]. Hypothesis 2: The longer feasibility study phase can lead to the better project estimation accuracy of resource. [Wiegers 03] Hypothesis 3: The longer feasibility study phase can decrease the probability of the project resource overrun [Hooks and Farry 01] Hypothesis 4: The lower requirement volatility can decrease the probability of project resource overrun [Zowghi & Nurmuliani 02] Hypothesis 5: The longer pre-study phase can save the project development time. 11

20 Hypothesis 6: The longer pre-study phase can reduce the development project lead time at each phase. Hypothesis 7: The longer pre-study phase can lead to the better project estimation accuracy of schedule, cost, or effort. Hypothesis 8: The longer pre-study phase can decrease the probability of project resource overrun. Hypothesis 9: The longer project development time can prolong the project time to market. 5.2 A SLDP Measurement Program A SLDP measurement program was based on the literature review mentioned and the data availability in the SLDP. This measurement program was also confirmed by project managers and supervisors. For example, the measurement on the project size was confirmed by project managers in the questionnaire. The following is the list of this SLDP measurement program. - main requirement lead time It is the time that a main requirement spent at the pre-study phase. (the QD-A date attribute, the QD-B date attribute, and the QD0 date attribute) - grouped main requirements lead time It is the time that grouped main requirement spent at the pre-study phase. (the earliest QD-A date attribute, and the QD0 date attribute) - main requirement size It can be measured by the number of test cases for each main requirement. - requirement volatility The number of the main requirement change requests and the total number of main requirements are recorded. The requirement volatility can be computed to be: added + modified+ deleted req. Requirement volatility = *100% [Wiegers 06] initial number of req. - project lead time It is the development time at each development phase. (QD0-QD1: the planing phase, QD1-QD2: the designing phase, QD2-QD3: the implementation phase, QD3-QD4: the functional testing phase, QD4-QD5: the non-functional testing phase, QD5-QD6: the latest system version phase, QD6-Released: the release phase) - project size It can be measured by the actual total man-hour of the project. 12

21 - project fault density The project fault density could be computed to be: number. of. fault fault. density = *100% [Mohagheghi et al. 04] project. size - completeness The assessment of the presence and agreement of all necessary software system parts. [IEEE 982.1] It can be computed to be: # output. req. final. report completene ss = *100% [Salamon & Wallace 94] # input. req. project. spec. - resource overrun We would compare the actual values with the estimated values of total effort, cost and schedule. If it is resource overrun, this attribute is assigned to be 1. Otherwise, this attribute is assigned to be 0. [Zowghi & Nurmuliani 02] - estimation accuracy (effort, cost, and schedule) To measure the estimation accuracy, there are five steps [Fenton & Pfleeger 98]: 1. The relative error (RE) is computed to be: Actual Estimate RE = (If the relative error is negative, then it is Actual overestimated. If the relative error is positive, then it is underestimated.) 2. The magnitude of the relative error (MRE) is computed to be: Actual Estimate MRE = (i.e. MRE is the absolute value of the relative error) Actual 3. The mean relative error for n projects is computed to be: n 1 RE = RE i n i= 1 4. The mean magnitude of relative error is computed to be: n 1 MRE = MRE i n i= 1 5. The estimation accuracy is computed to be: PRED ( q) = k / n (where q is assigned to be 0.25, k is the number of item whose magnitude of the relative error is less than q, and n is the number of projects). Usually, the estimation technique is acceptable if PRED(0.25) is at least

22 5.3 Data Collection In this study, a document inspection was used as a main way to collect data. A questionnaire, an interview and some specific s were also used to complement some missing data and investigate the reasons. The source of data is presented in AppendixⅠ. - Database There are two requirement management tools at Ericsson. One is used in the product management from the high level perspective. It contains all the approved or rejected main requirements, an overview of each development project and each release project. The other one is used in the development project. We used one documentation system to get the documents and the archival records. The problem is that we do not have access to the high level requirement management system. The solution is that we asked for help from our supervisor at Ericsson, so that we can get some information from the system. - Document inspection To examine how the requirements were gone through the whole development process, the document inspection was a main way in this study. We inspected the documents and the archived records. - Questionnaire To complement the missing data in the measurement program, a questionnaire was respectively sent to four project managers. They were responsible for those seven projects. The questionnaire is in the AppendixⅡ. - Interview and s To investigate the reasons, an interview was conducted in Sweden. We also used the other ways (such as s or instant messenger) to contact with project managers, because other three project managers are in China. 5.4 Data Analysis In this study, there are two parts of the data analysis. One is the data analysis on the SLDP measurement program, the other is the data analysis on the hypotheses - the correlations between requirement attributes and process attributes. 1. Data analysis on the SLDP measurement Microsoft Excel 2003 was used for the data visualization: a Gantt chart, a column chart, a pie chart, and a line chart. The average value and standard deviation were also computed in the Excel. 2. Data analysis on the Hypotheses testing Two statistical software applications (STATGRAPHICS Centurion XV version and StatsDirect version 2.6.2) were used in the hypotheses testing. 14

23 At first, the STATGRAPHICS Centurion was used to calculate the standardized skewness and the standardized kurtosis. Those two values can be used to determine whether the data is from the normal distribution. If those two values are in the range of -2 to +2, it indicates that the data are from the normal distribution [Users Guide in STATGRAPHICS Centurion]. It is suggested that the Pearson correlation coefficient techniques can be used in the normal distribution of data, and the Kendall ranking correlation coefficient can be used in the non-normal distribution of data [Fenton & Pfleeger 98]. The Pearson correlation coefficient technique was used in the STAGRAPHICS Centurion. The Kendall ranking correlation coefficient technique was used in the StatsDirect. The results of those hypotheses are presented by a correlation efficient r-value. This value is in the range of -1 to +1. If the correlation coefficient is closer to -1 or +1, it can indicate that the relationship between two variables is stronger. [Users Guide in STATGRAPHICS Centurion] According to [@Wikipedia], some authors suggested the guidelines for the interpretation of a correlation coefficient. Table 1 shows a guideline for the interpretation of a correlation coefficient from Cohen [@Wikipedia]. Table 1 A guideline for the interpretation of a correlation coefficient [@Wikipedia] Correlation Negative Positive small 0.29 to to 0.29 medium 0.49 to to 0.49 large 1.00 to to 1.00 Moreover, the P-Value is to determine whether two variables are significantly related, i.e. the level of statistical significance [Users Guide in STATGRAPHICS Centurion]. Usually, the P-Value is set 0.05 as the acceptable significance [Fenton & Pfleeger 98]. 5.5 Validation Construct Validity Four aspects were used to ensure the construct validity in this study: - Data triangulation [Yin 03]: More than one source of data was used. We inspected the documents and collected the archived records. A questionnaire was sent to four project managers who were the key informants in the projects. An interview and some specific s were also used to investigate the reasons. To solve the problem of data inconsistency, we contacted with the Ericsson supervisor and project managers. - Investigator Triangulation [Yin 03]: Two students were doing this study. Some follow-up questions were mentioned in the questionnaire, the interview, and some specific s. - Theory triangulation [Yin 03]: This SLDP measurement program was based on an extensive literature review. Four hypotheses on the correlations between the requirement attributes and the process attributes were derived from other researchers findings. The other five hypotheses were designed for the new requirement attributes and new process attributes in the SLDP. 15

24 - Methodological triangulation [Yin 03]: The multiple cases study was selected as a case study research strategy. Those seven projects are all completed projects developed in the streamline development process. A document inspection was used as a main way to collect data. A questionnaire, an interview and some specific s were also used to complement some missing data and investigate the reasons Internal Validity In this study, those measurements of requirement attributes and process attributes were based on the literature review. At least, those measurements are commonly used in the industry. Furthermore, those measurements were confirmed by project managers and supervisors. For example, the project size was confirmed by four project managers in the questionnaire. The applicability of requirement measurements was thus ensured. When it comes to those nine hypotheses, four hypotheses were from the current research findings on the correlation between requirement attributes and process attributes. The other five hypotheses were derived from the characteristics of the SLDP: the main requirement analysis at the pre-study phase and at the feasibility study phase, the separation between the latest system version phase and the release phase External Validity As we mentioned before, this SLDP has just been applied since late Those seven projects were all from the same product line. Some projects were developed in sequence and some projects were developed in parallel. Those seven projects were developed during Thus, the findings based on those seven projects can represent the general situation of SLDP at Ericsson AB Reliability To ensure the replication of this study, a report of this study and a case study database (Excel file) were documented. Thus, this SLDP measurement program and the correlations between requirement attributes and process attributes can be applied to other products in the streamline development process, if the data is available. 16

25 6 MEASUREMENT RESULTS This chapter would present and analyze the result of the SLDP measurement program mentioned in Chapter Main Requirement Lead Time - main requirement lead time (MRLT) It is the time that a main requirement stays in the pre-study phase. (the QD-A date attribute, the QD-B date attribute, and the QD0 date attribute) At Ericsson, there are three QD assessments in the pre-study: - QD-A: the main requirements are approved - QD-B: the business decision is made and the system analysis is initiated - QD0: the development project scope is defined and the feasibility study is started. There are two phases in the pre-study phase: the requirement definition phase (QDA- QDB) and the system analysis phase (QDB-QD0). In the requirement definition phase, the main requirement is analyzed from the business perspective. In the system analysis phase, the main requirement is analyzed from the technical perspective. Figure 4 shows the main requirement lead time at the requirement definition and at the system analysis phase. A B C D E F G QDA-QDB QDB-QD0 Figure 4 The main requirement lead time As it is shown in Figure 4, each main requirement is started on different dates. It can be caused by the issued date of a main requirement. Then, the main requirements are analyzed and prioritized from the business perspective. There are four types of prioritized main requirements in the requirement repository: current, planned, product customization, and future. Based on the requirement priority list, the main requirement is passed to the system analysis phase. 17

26 This main requirement lead time at the two phases cover the time spent on making the business decision, the effort on the main requirement analysis, and the waiting time for resource allocation. The variance at those two phases is very high. 6.2 Main Requirement Size - main requirement size It can be measured by the number of test cases for each main requirement. At Ericsson, testers estimate the number of test cases for each main requirement in a development project. Figure 5 shows the type of the main requirement in this study. It is shown that the number of test cases for each main requirement varies a lot. This could be caused by the different type of main requirement, the requirement complexity, or testers experience. functional req. a summary req. deleted req. non-functional req. packaged req. packaged req. Figure 5 The main requirement size Those main requirements can be categorized into five types: 1. A functional requirement: testers designed the functional test cases. 2. A summary requirement: this main requirement included other main requirements. There was no test case for this type of requirement. 3. A deleted requirement: this main requirement was removed when one trouble report solved this function. 4. A non-functional requirement: testers design the non-functional test cases. 5. A packaged requirement: two main requirements are packaged together. Thus, testers design the functional test cases for the packaged requirement. 18

27 6.3 Requirement Volatility - requirement volatility The number of the main requirement change requests and the total number of main requirements are recorded. The requirement volatility can be computed to be: added + modified+ deleted req. Requirement volatility = *100% initial number of req. [Wiegers 06] There are no change requests recorded in the requirement management systems and the document. Thus, a questionnaire was sent to four project managers, who were responsible for those seven projects. Only one project manager said that there was a main requirement added during the feasibility study phase. The other project managers all replied that there was no change in the projects. Table 2 shows the requirement volatility of those seven projects. The reason for the low requirement volatility was that the requirement changes from the project manager s view were only related to the features of a product. Table 2 The requirement volatility project requirement volatility A 0 B 0 C 0 D 0 E 0 F 14.29% G 0 We also did the document inspection for this requirement attribute. There were some records in the documents with regards to changes. First, some main requirements were packaged together during the designing phase. Second, the project scope was reduced or expanded during the feasibility study phase. 6.4 Project Lead Time - project lead time It is the development time at each development phase. (QD0-QD1: the planning phase, QD1-QD2: the designing phase, QD2-QD3: the implementation phase, QD3-QD4: the functional testing phase, QD4-QD5: the non-functional testing phase, QD5-QD6: the latest system version phase, QD6-Released: the release phase) At first, we calculated an average lead time between the grouped QD, because some projects grouped the QD assessment (QD0-QD2, QD2-QD5, QD5-QD6, and QD6- Released). Figure 6 shows the average lead time and the standard deviation between the grouped QD. From this graph, the release phase (QD6-Released) is spent the longest phase, and the variance is the highest. The following section would present the detailed information at each development phase. 19

28 the number of days QD0-QD2 QD2-QD5 QD5-QD6 QD6-Released average standard deviation Figure 6 The average lead time and the standard deviation between the grouped QD The feasibility study phase The feasibility study phase is composed of two phases: the planning phase (QD0-QD1) and the designing phase (QD1-QD2). Since two projects (project E and project F) grouped QD1 and QD2 assessment, five projects were used in this analysis. Figure 7 shows that the planning phase takes longer time than the designing phase. Moreover, the variance of the planning phase is higher than the designing phase. the number of days QD0-QD1 average QD1-QD2 standard deviation Figure 7 The average lead time and the standard deviation at the feasibility study phase During the document inspection, we found that the QD2 assessment was always delayed. Figure 8 shows the reasons for the delay at QD2. The main reason could be caused by an unapproved project budget, an unapproved project assignment or an unapproved delivery plan, which should have been ready at QD1. The other reason is that the main requirement was required to be clarified in more detail for the design team. At last, the other reason is that the test feasibility report had not been finished. 20

29 a project budget, assignment,or delivery plan is not approved a main requirement is not clear enough the test feasibility report is ongoing Figure 8 The reasons for the delay at QD2 assessment The implementation and testing phase Two projects (project E and project F) also grouped the QD2-QD5 assessment, and thus five projects were investigated in this analysis. Figure 9 shows the average lead time and the standard deviation at the implementation phase (QD2-QD3), the functional testing phase (QD3-QD4), and the non-functional testing phase (QD4-QD5). These three development phases have the same standard deviation value. the number of days QD2-QD3 QD3-QD4 QD4-QD5 average standard deviation Figure 9 An average lead time at the implementation and testing phase We also found the QD5 assessment was passed with some exemptions. Figure 10 shows the deviations at QD5. Besides the left trouble reports, the testing code generated by the testing tool was not fully verified, the uncompleted functional testing and system testing, the uncompleted non-functional requirement, the unapproved Background Activities and Traffic (BAT) specification, and the uncompleted test report were all the exemptions at the QD5 assessment. 21

30 some TRs were not closed Testing code was not be fully verified uncompleted FT and ST a non-functional requirement is fulfilled with deviation. unapproved BAT specification, uncompleted test report Figure 10 The deviations at QD5 assessment The latest system version phase The latest system version phase (QD5-QD6) is the end of project development. As it is shown in Figure 6, this phase is the shortest phase. However, some projects were also delayed at QD6. There are two main reasons led to the delay at QD6: - The uncompleted testing was handled to the design maintenance team, and this work was in parallel with the latest system version testing. - The merge issue and the thorough regression testing have to be done in this phase The release phase The release phase refers to the time between QD6 and Released. Figure 11 shows the lead time at QD6-Released. As it is shown in the graph, the project time to market varies a lot. A B C D 0 E F G QD6-Released Figure 11 A project time to market 22