Estimation of Reliability and Cost Relationship for Architecture-based Software

Transcription

1 International Journal of Automation and Computing 7(4), November 2010, DOI: /s Estimation of Reliability and Cost Relationship for Architecture-based Software Hui Guan 1, 2 Wei-Ru Chen 1 Ning Huang 1 Hong-Ji Yang 2 1 Department of Computer Science and Technology, Shenyang Institute of Chemical Technology, Shenyang , PRC 2 Software Technology Research Laboratory, De Montfort University, Leicester LE19BH, UK Abstract: In this paper, we propose a new method to estimate the relationship between software reliability and software development cost taking into account the complexity for developing the software system and the size of software intended to develop during the implementation phase of the software development life cycle. On the basis of estimated relationship, a set of empirical data has been used to validate the correctness of the proposed model by comparing the result with the other existing models. The outcome of this work shows that the method proposed here is a relatively straightforward one in formulating the relationship between reliability and cost during implementation phase. Keywords: Development cost, failure rate, software architecture, software reliability. 1 Introduction Software systems have a significant influence on our daily lives. A failure during operation of software can lead to economic loss and may even cause loss of human lives. Therefore, unreliable software is not acceptable and should be identified in the early stage of software development. The later the defects found, the higher the cost that needs to be paid for them. Reliability is one of the most important quality attributes of software since it quantifies software failures during the development process. However, higher level reliability of software can definitely result in more development cost which, in turn, restricts the software distribution [1]. It is concluded that a strong relationship exists between cost and reliability. In practical engineering, software project managers are required to estimate the cost needed to complete the development of a software system taking into account the required level of reliability. Indeed, the cost should be estimated, as early and accurately as possible, so that the development team can choose the proper techniques in order to address constraints of cost and required reliability. Moreover, estimating the development cost for a software system before being developed is an important foundation for the feasibility study and is the precondition to software system management. Therefore, after the design phase is completed, it is necessary to estimate the cost of implementation phase based on the architecture generated out of the design phase taking the required level of software reliability into consideration. Several software reliability models had been proposed in the past decades to help software managers and developers to analyze and design systems under such dual and often conflicting constraints of maximizing reliability and minimizing cost. In spite of there being many software reliability models, most of them focus on the cost estimation during the testing phase of software development [2 5] and software release policies [6 8], i.e., when to stop testing and Manuscript received July 8, 2009; revised January 10, 2010 deliver the system. We believe it is fair to say that there has been limited success as yet. Dating back to the last decade, Henlander et al. [9] described the relationship between reliability and cost based on reliability allocation approach. They presented two approaches for reliability and cost planning named reliability-constrained cost-minimization (RCCM) and budget-constrained reliability-maximization (BCRM) and three functions for measuring cost-to-attain failure intensity. The modeling frameworks presented in [9] focus on cost distribution among software components with a quantified system reliability goal. In addition, Zahedi and Ashrafi [10] adopted the analytic hierarchy process (AHP) method for modeling the software architecture with cost as the constraints and proposed a model relating to the system reliability maximization. Boehm et al. [11] presented a method for evaluating software development cost by using the constructive cost model (COCOMO) model. Similar to the COCOMO model, Forbes and Long [8] proposed a reliability investment model that can be used to predict the investment in reliability required to achieve a given amount of reliability improvement by dividing the investment in reliability in terms of average production unit cost. The model proposed in [8] is intended to comprise four submodels; only two of them are completed thus far, and more research is required to mature the basic model. Huang and Kuo [2, 3] outlined a method using logistic testing-effort (LTE) function to optimize the reliability allocation and testing schedule for a software system taking into account the reliability growth of its components. Wadekar and Gokhale [12] explored cost and reliability tradeoffs in architectural alternatives by using a genetic algorithm. Williams and Vivekanandan [13] discussed a modified approach to calculate the delivery cost of a software product, when warranty is to be provided, with an imperfect debugging phenomenon. The optimal release time can be calculated for various reliability levels by minimizing the cost. The delivery cost, reliability of the software system, and the optimal release time were calculated by using an imperfect debug-

2 604 International Journal of Automation and Computing 7(4), November 2010 ging software reliability growth model. There are many other different methods to address the relationship between software reliability and development cost [4, 14 16]. The contribution of this paper is to propose a new approach for estimating the cost of the implementation phase of the software development life cycle based on the software architecture, given a desired level of reliability for that software. Estimation is carried out after the design phase is conducted, that is, before the software implementation phase begins. Evaluation of the relationship of software reliability and development cost is based on the available architecture and the expected level of software reliability. An architecture-based reliability and cost relationship model which is in form of the logarithm of reliability (abbreviated as RCR model) is outlined, and then two comparisons between the proposed model and the models proposed by Helander et al. [9] are made via applying the empirical data. At last, the comparison results are given at the end of the paper. The rest of this paper is organized as follows. In Section 2, we introduce in detail the architecture-based reliability and cost relationship model, abbreviated as RCR. In Section 3, a numerical example is given in order to validate whether the RCR model can work properly. By comparison, the other three models proposed by Helander et al. [9] are provided in Section 4. In Section 5, we utilize two sets of empirical data to verify the correctness of the RCR model and make comparisons with the models mentioned in Section 4. In the end, conclusion and future work are presented. 2 Architecture-based reliability and cost relationship model It is reasonable to assume that the cost function f i would satisfy these three conditions [17] : 1) f i is a positive definite function. 2) f i is non-decreasing. 3) f i increases at a higher rate for higher values of R i. The third condition suggests that it can be very expensive to achieve the reliability value of 1. In fact, it has been shown that under some assumptions, it is infeasible to achieve ultra-high reliability for software [18]. In some cases, the cost function can be derived from basic considerations and is usually stated in terms of the reliability, as we will do below for software reliability. 2.1 Software reliability According to the definition of software reliability [19] : r(t) = e λt (1) where r(t) is the continuous-time system reliability, and λ is its failure rate. From the formula above, it can be concluded that the reliability of a software is strongly dependent on the number of faults that remain in it after testing and debugging. As fault decreases, the probability that the software works according to its specification will increase, that is, reliability will vary inversely with faults. 2.2 Development cost Software development process consists of several phases, where estimation of development cost means different things with respect to different phases of software development life cycle. Fig. 1 gives some examples of factors associated with development cost of each phase. However, it is not limited to these. Fig. 1 Cost in terms of software development life cycle In this paper, we focus on the development cost spent in the implementation phase of the software development life cycle. We define development cost as the cost incurred to develop the software system while meeting the required level of reliability based on the existing architecture which is completed during the design phase. 2.3 Two assumptions of the RCR model According to the experiences from practical engineering, this paper takes account of the factors by assuming that software reliability and development cost satisfy the relationships as below: Assumption 1. Software development cost is inversely proportional to the number of system failures. That is E 1 C (2) λ E (3) where E is the number of system failures, C is the development cost, and λ is the system failure rate. Using (2) and (3), reliability function (1) is therefore C 1 ln r. (4) From the above formulation, we can conclude that software development cost is inversely proportional to the logarithm of achieved or required reliability r without taking the running time into account. Assumption 2. Software development cost is proportional to the complexity and size of the software system. Taking Assumptions 1 and 2 into account, (4) can be approximated by C = α (5) ln r where α represents the complexity and size of developing the software system and is called the reliability-cost coefficient in this paper.

3 H. Guan et al. / Estimation of Reliability and Cost Relationship for Architecture-based Software 605 Consider the matter of fact in practical engineering, where there usually exists some basic cost such as personnel training, development tools preparation, etc., we use character β to represent the basic development cost in this paper. With the assumption above, we describe the reliabilitycost model as below: C(r) = α + β. (6) ln r Equation (6) states the relationship between the system development cost and the system reliability in our RCR model. 2.4 Considerations on α and β A software application that is developed in an ideal manner will go through a first round of architecture-based analysis during the design phase. When architecture-based analysis is conducted during the design phase, the architecture of the application may be obtained from expert opinion, from prior experience with similar product, and from the previous release of the same product. On this basis, estimation of architecture-based software development cost with the required level of software reliability can be conducted by using (6). We do not derive explicit expressions for α and β in this work; rather, we assume that α and β are estimated during the design phase, based on the historical data of past projects of the same category as the software under development. Parameter β is called basic development cost in this paper which includes the cost paid before software development, i.e., personnel training, software and proprietary technology purchasing, etc. Historical basic data of similar project may do help in the estimation of β. With regard to α called the reliability-cost coefficient in this paper, it has close relationship with the complexity and size of software. As far as a specific software project development team is concerned, we can estimate α from previous practical project data if we know the size and technology of developing the software; the value of α will be determined accordingly. For example, it is common for a software development team that a default level of reliability will be set before the software is being developed. Suppose that r 0 is the default reliability. C 0 can be estimated from the historical project of the same category for the same development team, and then α can be estimated by using (6). With regard to the basic indivisible software or component and a specified software development team, α and β are fixed values. 3 Numerical example of the RCR model Consider a hypothetical example where we wish to estimate the overall development cost C, taking into account the two parameters, which are β = 90 and α = 0.3; then we can obtain the data shown in Table 1 according to the formula of the RCR model. Fig. 2 shows the relationship between the system reliability and the development cost of the RCR model with the data in Table 1, where r and C refer to the software reliability and the development cost, respectively. Fig. 2 Curve of C and r for the hypothetical example Table 1 Reliability and cost data set with β = 90 and α = 0.3 No. r C According to Fig. 2, it will cost more in order to achieve a very high level of software reliability. Conclusion from the computation by using first optimization (1stOpt) tools is that our RCR model can express the relationship between software reliability and development cost preliminarily. Therefore, it may be said that the result produced here as expected has yielded a reasonable consistent estimation. 4 Three software component cost failure intensity functions Helander et al. [9] proposed three functions of software development cost according to the different relationships among the influence factors: linear, exponential, and inverse power. 4.1 The linear cost function The linear form (see Table 2) can be used in the absence of knowledge about the relative nonlinear characteristics of cost-to-attain reliability, or when this relationship is thought to be indeed linear. The advantage of using the linear cost function is its simplicity. The disadvantage of the linear function is that the solution might not be practical for large or complex systems.

4 606 International Journal of Automation and Computing 7(4), November 2010 Table 2 General forms and parameter specifications for software component cost-to-attain failure intensity functions Cost form Function Parameter αλ + β, if λ β β Linear C (λ) = α Scale β > 0, Shape α (0, ) λ 0, otherwise max β ln (1 exp ( λ)), if λ > 0 Logarithmic exponential (LnExp) C (λ) = +, otherwise Scale β > 0 Inverse power (InvPow) C (λ) = β (λ δ) α, if λ > δ +, otherwise Scale β > 0, Shape α 0, Location δ (0, λ s) 4.2 The logarithmic exponential cost function (LnExp) A second cost function, which is nonlinear in form, is the logarithmic exponential (LnExp) function. An advantage of the LnExp cost function is that only a single parameter β needs to be specified for each component. Also, in the absence of actual data, approximation of the single parameter β and visual comparison of the resulting curves to assess and verify relative software component costs are possible. A disadvantage of the LnExp function is that there is little flexibility for changing the shape of the curve. 4.3 The inverse power cost function (InvPow) Another nonlinear function that satisfies the desired characteristics is the inverse power (InvPow) cost function. The advantage of the InvPow cost function is that it is a generalization of the basic COCOMO form introduced by Boehm [11]. A disadvantage of the InvPow cost function is that either two or three parameters per component must be specified, depending on whether the location parameter δ is assumed to be nonzero. 5 Comparisons and results In this section, we illustrate the applicability of the RCR model with two examples by using two sets of empirical data. In fact, it is difficult to obtain the statistical data in terms of cost and software reliability. Unfortunately, the data listed in the previous literatures are mostly based on assumption. We refer to the data cited from [20, 21], both of which are from architecture-based software. Statistical data from [20] are derived from a practical project named post logistics information system of Jiangsu Province. A curve fitting tool named 1stOpt is used in this paper. 1stOpt is developed by 7D Software High Technology Incorporation. It is a leading worldwide software platform for numerical optimization analysis, especially in areas of curve fitting, nonlinear regression and parameter estimation of a nonlinear complex engineering model. We use this tool as the experimental basis. 5.1 Example 1 This example illustrates the correctness of the RCR model when it is applied by using empirical data cited from [20]. In order to make a comparison with other reliabilitycost models, the same data are applied to the models proposed by Helander et al. [9] as well. In [22], when the software reliability is 0.93 and development cost is 90, we can obtain Table 3. The meaning of each symbol is the same as in (5), where λ represents the failure ratio, and r and C are software reliability and development cost, respectively. Table 4 shows the comparison results of four cost models listed in Sections 3 and 4 by using the empirical data of Table 3, and their comparison plots are shown in Fig. 3. Table 5 depicts the best estimated value of parameters for each cost model. Table 3 Relationship between failure rate and reliability-cost within unit time λ r C In Fig. 3, small dot line with black solid circles represents the calculated value, and solid lines with black solid rectangles represent the actual value with the reliability on horizontal axis and development cost on vertical axis in the RCR model, and the failure rate λ on horizontal axis and development cost C on vertical axis in the other three figures. The trends of the calculated value curve and that of the actual value curve of four models are all consistent. 5.2 Example 2 Let us consider another example where the RCR model is used to estimate the additional development cost when the system reliability is required to be improved on the basis of the reliability available.

5 H. Guan et al. / Estimation of Reliability and Cost Relationship for Architecture-based Software 607 Table 4 Comparison results of Example 1 Actual value RCR model Linear cost model LnExp cost model InvPow cost model Calculated value Calculated value Calculated value Calculated value Average comparative error Table 5 Best estimated values of each parameter for the four cost models Example Example 1 Example 2 Cost model Parameter α β δ RCR model Linear cost model LnExp cost model InvPow cost model RCR model LnExp cost model (a) RCR model (b) Linear cost model (c) LnExp cost model (d) InvPow cost model Fig. 3 Fitted figures of four cost models with the empirical data in [18]

6 608 International Journal of Automation and Computing 7(4), November 2010 Suppose, for the system reliability available of 0.93, it will cost more if we intend to improve the reliability to a higher level. Some empirical data [21] are shown in Table 6. Table 6 Data in terms of reliability [20] Available r Expected r C(0.93, r) RCR model As the empirical data in Table 6 are in terms of the differentials of the cost, we need to transform (6) in order to be in conformity with empirical data; the result is C(0.93, r) = α ln r + β ( α ln β) = α ln r + α ln 0.93 We use (7) as the objective function in 1stOpt tool, where r is the expected level of reliability, 0.93 is the available system reliability and C(0.93, r) is the difference value of system development cost between them. Fig. 4 shows the trends of two curves using 1stOpt tool. (7) As the logarithmic-exponential cost model is expressed in the form of the system failure rate λ, whereas the empirical data [21] are given in terms of reliability, a transformation needs to be done. Table 7 lists the empirical data in the form of system failure rate λ. When the system available reliability r is 0.93, the corresponding system failure rate λ is by using (1). Table 7 Data in terms of failure rate [20] Available λ Expected λ C(0.0726, λ) With the same operation done to the RCR model, Fig. 5 shows the result of applying empirical data [21] to the logarithmic-exponential cost model. Fig. 5 Logarithmic-exponential cost model with data in Table 7 Table 8 Comparison result of Example 2 Actual value RCR model Calculated value LnExp cost model Calculated value Fig. 4 RCR model with empirical data in Table Logarithmic-exponential cost model In this case, we choose the logarithmic-exponential cost model from [9] as the compare model for the RCR model as both models are in the forms of logarithm. The same as in Section 5.2.1, we need to transform the logarithmic-exponential cost model as well. The transformed equation is ( ) (1 exp ( λ)) C(0.0726, λ) = β ln. (8) (1 exp ( )) Average comparative error

7 H. Guan et al. / Estimation of Reliability and Cost Relationship for Architecture-based Software 609 Comparison results are listed in Tables 4 and 8 for Examples 1 and 2. In the meanwhile, the estimated values of each parameter for both examples are given in Table 5. It can be concluded from the examples above that the values of the RCR model are more close to the actual value than the other three models proposed in [9] as far as the empirical data in Tables 3 and 6 are concerned. In addition, the RCR model is more straightforward, effective and needs less calculation. 6 Conclusion and future work In this paper, we formulate an estimation approach to model the relationship between software reliability and software development cost taking into account the complexity for developing the software system and the size of software intended to develop. On the basis of this, a set of empirical data has been used to validate the correctness of the RCR model by comparing the result with the other three models proposed by Heland et al. [9] The outcome of this work, despite being still preliminary, allows us to say that the method proposed here is a relatively straightforward one in formulating the relationship between reliability and cost during design phase. We present the RCR model with an eye on the complexity for developing the software system and the size of software intended to develop. However, how to make sure about the quantification of the parameters (α, β) still deserves further exploration. Furthermore, the RCR model can be used to estimate the cost of an independent component in terms of its allocated reliability. When it is used in such circumstance, how to allocate the limited development cost to the components while achieving the highest level reliability or allocate the reliability to the components with assuring required reliability of the whole software while minimizing the amount of the development cost is another problem we will try to study. References [1] W. Ning, Y. Chen, X. Tian. A balance between software reliability and cost. Systems Engineering and Electronics, vol. 24, no. 11, pp , (in Chinese) [2] C. Y. Huang, J. H. Lo, S. Y. Kuo, M. R. Lyu. Optimal allocation of testing resource considering cost, reliability, and testing effort. In Proceedings of Pacific Rim Dependable Computing, Polynesia, Frence, pp , [3] S. Y. Kuo, C. Y. Huang, M. R. Lyu. A framework for modeling software reliability, using various testing efforts and fault detection rates. IEEE Transactions on Reliability, vol. 50, no. 3, pp , [4] C. Y. Huang, J. H. Lo, S. Y. Kuo, M. R. Lyu. Software reliability modeling and cost estimation incorporating testing-effort and efficiency. In Proceedings of the 10th International Symposium on Software Reliability Engineering, IEEE, pp , [5] H. Pham, H. Wang. A Quasi-renewal process for software reliability and testing costs. IEEE Transactions on Systems, Man, and Cybernetics Part A: Systems and Humans, vol. 31, no. 6, pp , [6] C. Y. Huang, M. R. Lyu. Optimal release time for software systems considering cost, testing-effort, and test efficiency. IEEE Transactions on Reliability, vol. 54, no. 4, pp , [7] B. Yang, H. Hu, J. Zhou. Optimal software release time determination with risk constraint. In Proceedings of Annual Reliability and Maintainability Symposium, IEEE, pp , [8] J. Forbes, E. A. Long. Dertermining how much to invest in reliability. International Test and Evaluation Association, vol. 29, no. 3, pp , [9] M. E. Helander, M. Zhao, N. Ohlsson. Planning models for software reliability and cost. IEEE Transactions on Software Engineering, vol. 24, no. 6, pp , [10] F. Zahedi, N. Ashrafi. Software reliability allocation based on structure, utility, price and cost. IEEE Transactions on Software Engineering, vol. 17, no. 4, pp , [11] B. Boehm, R. Valerdi, J. A. Lane, W. Brown. COCOMO suite methodology and evolution. CrossTalk The Journal of Defense Software Engineering, vol. 18, no. 4, pp , [12] S. A. Wadekar, S. S. Gokhale. Exploring cost and reliability tradeoffs in architectural alternatives using a genetic algorithm. In Proceedings of the 10th International Symposium on Software Reliability Engineering, IEEE, pp , [13] D. R. P. Williams. Study of the warranty cost model for software reliability with an imperfect debugging phenomenon. Turkish Journal of Electrical Engineering and Computer Sciences, vol. 15, no. 3, pp , [14] N. E. Rallis, Z. F. Lansdowne. Reliability estimation for a software system with sequential independent reviews. IEEE Transactions on Software Engineering, vol. 27, no. 12, pp , [15] O. Hryniewicz. An evaluation of the reliability of complex systems using shadowed sets and fuzzy lifetime data. International Journal of Automation and Computing, vol. 3, no. 2, pp , [16] B. Kwiatuszewska-Sarnecka. Reliability improvement of large multi-state series-parallel systems. International Journal of Automation and Computing, vol. 3, no. 2, pp , 2006.

8 610 International Journal of Automation and Computing 7(4), November 2010 [17] A. Mettas. Reliability allocation and optimization for complex systems. In Proceedings of Annual Reliability and Maintainability Symposium, Los Angeles, CA, USA, pp , [18] R. W. Bulter, G. B. Finelli. The infeasibility of quantifying the reliability of life-critical real-time software. IEEE Transactions on Software Engineering, vol. 19, no. 1, pp. 3 12, [19] M. R. Lyu. Handbook of Software Reliability Engineering, New York, USA: IEEE Computer Society Press, pp. 36, [20] B. Han. Reliability Allocation and Applied Research of Software System Based on Genetic Algorithm, M. Sc. dissertation, Shanghai Normal University, PRC, (in Chinese) [21] W. Zhu, X. Shao, H. Hu, Y. Dong. Optimal decision for the allocation of system reliability. Journal of Systems Engineering, vol. 13, no. 1, pp ,1998. (in Chinese) [22] K. Kapur, L. R. Lamberson. Reliability in Engineering Design, New York, USA: John Wiley and Sons, Inc., Hui Guan received the B. Sc. and M. Sc. degrees in computer science from the Shenyang Institute of Chemical Technology, PRC in 2000 and 2006, respectively. Since January 2009, she has been a Ph. D. candidate at De Montfort University, UK, in the Faculty of Computing Science and Engineering. In the meantime, she is a lecturer in the Shenyang Institute of Chemical Technology. Her research interests include software reliability, model driven development, and software security. guanh1999@126.com (Corresponding author) Wei-Ru Chen received the B. Sc. and M. Sc. degrees from Northeast University, PRC in 1985 and 1988, respectively. He is currently the head of the Department of Computer Science and Technology, Shenyang Institute of Chemical Technology, PRC. His research interests include software architecture, software reliability, and data mining. willc@china.com Ning Huang received the B. Sc. and M. Sc. degrees in computer science from Shenyang Institute of Chemical Technology, PRC in 2003 and 2008, respectively. Now, she is a laboratory assistant in Shenyang Institute of Chemical Technology. Her research interests include software reliability and software architecture. huangning@syict.edu.cn Hong-Ji Yang received the B. Sc. and M. Phil. degrees from Jilin University in 1982 and 1985, respectively, and the Ph. D. degree from Durham University, UK in Currently, he is a professor at the Software Technology Research Laboratory, Faculty of Technology, De Montfort University, UK and leads the Software Evolution and Reengineering Group. He served as a program co-chair at IEEE International Conference on Software Maintenance in 1999 and the program chair at IEEE Computer Software and Application Conference in His research interests include software engineering and pervasive computing. hyang@dmu.ac.uk