Chapter 5 TOPVEC: A Critical Success Framework for COTS Based Software Development

Transcription

1 Chapter 5 TOPVEC: A Critical Success Framework for COTS Based Software Development

2 CHAPTER 5 TOPVEC: A CRITICAL SUCCESS FRAMEWORK FOR COTS BASED SOFTWARE DEVELOPMENT The face of software industry is changing rapidly and there s no stopping it. Software industry is undergoing a number of fundamental shifts due to current economic necessities and ongoing technological developments. In order to keep pace with the ever newer requirements and remain relevant to their clients and customers, it is increasingly important for software development companies to develop software products that are reliable, cost effective and delivered on time. Now a days software are no more developed using the traditional software engineering approach, rather they are built by assembling the pre-packaged solutions, usually known as Commercial-off-the-Shelf (COTS) components. This approach of software development is known as COTS Based Software Development (CBSD). CBSD has proved an economic and strategic way of developing large and complex system. This chapter seeks to identify and provide insight into the critical success factors (CSF s) that help CBSD to succeed. The objective of this chapter is to propose a conceptual framework consisting of the key factors influencing the success of projects based on CBSD approach. A web-based survey was conducted to gather feedback from 115 software professionals involved in CBSD. This chapter compiles the success factors reported in CBSD literature, performed reliability analysis and factor analysis on those factors. They are further consolidated into final 7 possible success factors for CBSD projects in five different categories: Technical, Organizational, People, Vendor Selection, and Cost, and hence the proposed framework known as TOPVEC. The data is analyzed using multiple regression techniques.2 This chapter is based on the paper given below: Bali, V. and Madan, S. (2015), "A Study of Critical Success Factors in COTS Based Software Development: TOPVEC Framework", Wulfenia Journal, Vol. 22, No. 8, pp , SCI Impact Factor: Vikram Bali, Ph.D. Thesis

3 The analysis done in this chapter addresses the following research questions: 1. Are the identified factors truly, critical success factors of COTS based software development? 2. If so, what is the relative importance of each factor when compared to other factors? 3. Is there a significant difference among those five factor categories in terms of their impact on the success of COTS based software project? The objective of this chapter is to perform reliability analysis and factor analysis to consolidate the listed factors into a final set of seven possible critical success factors for each of the four project success dimensions Quality, Scope, Time and Cost. The final list of seven factors is depicted by the proposed framework known as TOPVEC. Multiple regression technique is used for analysis of data and the results reveal that only 17 hypotheses out of 28 are supported which leads in identifying four critical success factors for COTS software development projects: (a) COTS software development techniques (b) healthy organizational environment (c) Vendor evaluation strategy, and (d) pricing strategy. 5.1 BACKGROUND In this section, two key concepts are discussed in brief. COTS based Software Development (CBSD) and Critical Success Factors (CSF) COTS Based Software Development The term commercial off-the-shelf (COTS) is very generic; it can refer to many different types and levels of software, e.g., software that provides a specific functionality or a tool used to generate code. COTS may be one of the most diversely defined terms in current software development [Morisio et al., (2000)]. Vikram Bali, Ph.D. Thesis

4 A COTS product can be defined [Comella et al., (2004)] as the one that is Sold, leased, or licensed to the general public Offered by a vendor trying to profit from it Supported and evolved by the vendor, who retains the intellectual property rights Available in multiple, identical copies Used without modification of the internals These COTS components can be developed by different developers using different languages and different platforms. Today s need for rapid software development has generated a great interest in employing CBSD approach for building a software system by assembling different components with a well-defined software architecture. This approach is very different from the traditional approach in which software systems can only be implemented from scratch. CBSD is defined as any system partially or completely constructed, using COTS software products as integral components. CBSD can lower the level of complexity by breaking down a large project into more manageable smaller pieces with the aim of reassembling them later in order to achieve the desired solution [Cheesman and Daniels, (2001)]. Component-based software development (CBSD) can significantly reduce development cost and time-to-market, and improve maintainability, reliability and overall quality of software systems [Pour et al., (1999)]. This approach has raised a tremendous amount of interest both in industry and academia. The SDLC and software engineering model of CBSD is different from that of the traditional ones. Six Steps for successful COTS implementation given by Minkiewicz et al., (2005) are given in Figure 5.1.The authors in their work highlighted that these six steps are important activities that should take place during a commercial off-the shelf (COTS) implementation. Following this methodology throughout the software development life cycle will ensure that significant activities are not being ignored and will increase the chances of planning, executing, and deploying a successful COTSbased software solution. Vikram Bali, Ph.D. Thesis

5 Figure 5.1: Six Steps for Successful COTS Implementation This study includes all the six stages of COTS based software solution to obtain the critical success factors that lead to its success The Critical Success Factor Approach Literature suggests that 90% of Information Technology projects fail to meet their goals, 80% are over-budgeted and 40% are discarded [Leibowitz, (1999)]. The Standish Group did a lot of work in 1995 and proposed that 31% of projects were cancelled before completion and 52.7% of the projects completed were over by 189% of the allocated budget [Leibowitz, (1999)]. Information Technology is the backbone of today's era. It is highly desirable and important that the implementations of the IT projects are done with utmost care. Researchers suggest that there are many factors which are important for the success of IT projects. These factors are known as Critical Success Factors (CSF). Determining CSFs for the software development projects are of great importance in today's scenario. Managers, developers and any other authority can use them for fulfilling their goals and to achieve desired result i.e. software which is acceptable to the customer and is developed by utilizing optimal resources, hence, increasing the bottom line of the Organization. According to recent studies, it is highlighted that the concept of critical success factor is of utmost importance in software development. Reel has proposed that CSFs in software projects are related to project management techniques [Reel, (1999)] whereas Bytheway has mentioned that it is combination of business strategy and software engineering [Bytheway, (1999)]. Further in this direction, Bosghossian, 2002 Vikram Bali, Ph.D. Thesis

6 suggested that the CSFs of software projects is a combination of different factors comprising from software development life cycle i.e validation, estimation, resource management, executive management and project management. Chow and Cao, (2008) in their study identified critical success factors of agile software development projects using quantitative approach. An extensive literature survey was done by Nasir and Sahibuddin, (2011) and then found that the factors of clear and frozen requirements, realistic estimation of the schedule and budget, along with a competent project management are the five most CSFs for software project. 5.2 SUCCESS FACTORS FOR COTS BASED SOFTWARE PROJECTS There has not been any formal study on critical success factors of COTS based software projects in the past literature. However, there are lots of research papers which talk about the important activities that must take place during software development using COTS based approach. Moriso et al. (2002) in their work identified product evaluation, product familiarization and vendor interaction as the important activities for COTS based processes. Minkiewicz, (2005) highlighted the important activities that should take place during COTS implementation. He also suggested that following these activities will increase the chance of planning, executing and deploying COTS based software solution. A great deal of research has been done in the area of COTS evaluation [Kontio, (1995); Ochs et al. (2000); Chung and Cooper (2004a); Tarawneh et al. (2011) and Mead (2014)] and selection [Berman and Kumar (1999); Kapur et al. (2003); Cortellessa et al. (2008); Jha et al. (2010) and Bali et al. (2014)]. Based on the literature discussed, it can be safely stated that COTS based software projects can be classified into five categories: Technical, Organizational, People, Vendor Selection, and Cost, summarized in Table 5.1. Vikram Bali, Ph.D. Thesis

7 Table 5.1: List of Success Factors Identified from the Literature Dimension Organization People Vendor Selection 1. Strong management support Factor 2. Goals and objectives of the organization must be clearly defined 3. Well-defined organizational structure 4. Better to have cooperative organizational structure instead of hierarchical structure 5. Presence of organizational politics badly affect the employee s performance 6. Support software development by assembling pre-packaged (also known as COTS) components 7. Proper facility and work environment for its employees 8. Motivate the employees through proper reward system 9. Project teams with members having high competence and experience 10. Committed manager or team leader responsible for successful completion of the project 11. Team members have a coherent self-organizing behavior 12. Team members must work with great motivation in an organization 13. Customer s/client s regular involvement is essential during the software development process 14. Organization sensitive towards customer s expectations 15. Evaluate vendors on the basis of the parameter vendor stability 16. Evaluate vendors on the basis of the parameter vendor capability 17. Evaluate vendors on the basis of the parameter vendor reputation 18. Evaluate vendors on the basis of the parameter vendor s financial stability 19. Training to be given by the vendor about the pre-packaged software component 20. Evaluate vendor on the basis of after sales support 21. Evaluate vendors on the basis of the parameter vendor s support infrastructure 22. Delivery time of COTS product as important criteria for selecting them from vendor 23. Buy pre-packaged software from vendors who are actively involved in R&D activities 24. Market Analyst Perspective about the vendor is an important criteria for COTS vendor selection Vikram Bali, Ph.D. Thesis

8 Dimension Factor 25. Pricing of pre-packaged components 26. License fee of pre-packaged software components Cost 27. Modular pricing is the deciding factor 28. Maintenance fee of pre-packaged components play a significant role 29. Documentation fee is the deciding factor 30. Consultation fee of pre-packaged components 31. Robust and complete 32. Components compatibility issues be considered 33. Components integration issues must be considered 44. Final software product developed always close to the client s requirements Technical 35. Product which is delivered to the client should have shorter implementation time 36. Software system must be user friendly 37. Final product must be flexible to changes as per the business requirements 38. Software system must be adaptable 39. Product which is open to work with other systems 40. Need to issue a proper maintenance document to your clients The attributes of success related to software projects taken from the literature are mentioned in Table 5.2 Table 5.2: Success Dimension Dimension Overall perceived level of success Attribute 1. Quality (delivered a good product or outcome) 2. Scope (all requirements and objectives were met) 3. Time (delivered on time) 4. Cost (delivered within estimated cost and effort) Vikram Bali, Ph.D. Thesis

9 The data was collected through questionnaire which was distributed to the software professionals, working on COTS based software projects. To measure the importance of success factors, a seven point likert scale was used to reflect the level of perception of the question by the respondents. 5.3 DATA ANALYSIS AND RESULTS This research is of exploratory in nature. Therefore, a reliability analysis is performed to ensure a high level of reliability in all the factors. Reliability analysis provides an overall index of internal consistency of the variables, and helps in singling out the variable and/or included in another variable. Rubin and Babbie (1997) highlighted that the most common and powerful method used for calculating internal consistency and reliability is coefficient alpha. For exploratory studies, it is agreed that a coefficient alpha level of 0.5 shall be deemed acceptable [Nunally, (1967)]. A reliability analysis was performed on all the items using Cronbach s alpha method. The result of Cronbach s α test was 0.912, which is considered to be good. The closer the value of α goes to 1, the better is the reliability test [Cronbach, (1951)] Exploratory Factor Analysis A principal component factor analysis with varimax rotation was performed on the data items. Seven factors emerged from the analysis, accounting for 71.14% of the variance. Items with loadings 0.5 on the target construct were retained. The result of this analysis, details of items with factor loadings, and the reliability coefficients for each scale are given in the Table 5.3. Vikram Bali, Ph.D. Thesis

10 Table 5.3: Result of Factor Analysis Factors Factor 1: COTS Software Development Techniques Factor Loadings Factor 1 Factor 2 Factor 3 Factor 4 Factor 5 Factor 6 Factor 7 Communality Software system must be adaptable Components compatibility issues be considered Components integration issues must be considered Product which is open to work with other systems Factor 2: Pricing Strategy License fee of pre-packaged software components Maintenance fee of pre-packaged components play a significant role Modular pricing is the deciding factor Factor 3: Vendor Evaluation Strategy Evaluate vendor on basis of after sales support Evaluate vendors on the basis of the parameter vendor s support infrastructure Delivery time of COTS product as important criteria for selecting Vendor Factor 4: Healthy Organizational Environment Well-defined Organizational Structure Strong management support Goals and objectives of the organization must be clearly defined Factor 5: High Caliber Team Capability Team members have a coherent self-organizing behavior Team members must work with great motivation in an organization Project teams with members having high competence and experience Committed manager or team leader responsible for successful completion of the project Vikram Bali, Ph.D. Thesis

11 Factors Factor Loadings Factor 6: Strong Customer Involvement Organization sensitive towards customer s expectations Customer s/client s regular involvement is essential during software development process Factor 7: Vendor's Reputation Market Analyst Perspective about the vendor is an important criteria for COTS vendor selection Evaluate vendors on the basis of the parameter vendor reputation Eigen value % Variance Explained Cumulative variance (%) The seven factors given above were translated into 7 main hypotheses, each linking its existence as a critical success factor to the success of software developed based on COTS components, in terms of four success dimensions: Quality, Scope, Time, and Cost. The hypotheses were numbered 1 to 7, and since there were four success dimensions for each factor, the corresponding success dimensions for each factor were identified by letter a, b, c, d. As a result there were a total of 28 hypotheses; starting at 1a and ending at 7d. The hypotheses formulated for the seven factors, each addressing four success dimensions, are presented as follows: Technical Factor H1: The presence of COTS software development technique is a critical success factor that contributes to the success of COTS based software projects in terms of (a) Quality, (b) Scope, (c) Time, and (d) Cost Vikram Bali, Ph.D. Thesis

12 Organizational Factor H2: Healthy organizational environment is a critical success factor that contributes to the success of COTS based software projects in terms of (a) Quality, (b) Scope, (c) Time, and (d) Cost People Factor H3: High caliber team capability is a critical success factor that contributes to the success of COTS based software projects in terms of (a) Quality, (b) Scope, (c) Time, and (d) Cost H4: Strong customer environment is a critical success factor that contributes to the success of COTS based software projects in terms of (a) Quality, (b) Scope, (c) Time, and (d) Cost Vendor Selection Factor H5: Vendor evaluation strategy is a critical success factor that contributes to the success of COTS based software projects in terms of (a) Quality, (b) Scope, (c) Time, and (d) Cost H6: Vendor reputation is a critical success factor that contributes to the success of COTS based software projects in terms of (a) Quality, (b) Scope, (c) Time, and (d) Cost Cost Factor H7: Pricing strategy is a critical success factor that contributes to the success of COTS based software projects in terms of (a) Quality, (b) Scope, (c) Time, and (d) Cost. The final list of 7 factors is depicted by the proposed framework TOPVEC and is given in Figure 5.2. Vikram Bali, Ph.D. Thesis

13 Chapter 5: Figure 5.2: Proposed Framework - TOPVEC Multiple Regression Models This research is an exploratory study to find out which factors can positively impact the success of COTS based software development. A multiple regression analysis is performed, where the relationship between multiple independent variables (success factors) and the dependent variable (COTS project success) is determined, and the relative predictive importance of the independent variables is established [Williams and Monge, (2001)]. The multiple regression model with k independent variables can be written as follows: Y = β0 + β1 x1 + β2 x βk xk + ε Where Y is the dependent variable and x1, x2,..., xk are independent variables, and βi is the regression coefficient, and ε is the random error component. The value of the coefficient βi determines the contribution of the independent variable xi, given that the other x variables are constant and β 0 is the y -intercept. Vikram Bali, Ph.D. Thesis

14 For this work the above translates into the following general equation: Y ( Q, S, T, C) = β SF+ β SF β SF Where Y is the dependent variable that denotes the success of COTS based software project, Q is the Quality Dimension, S is the Scope Dimension, T is the Time Dimension, C is the Cost Dimension, th i Success Factor (SF). βi is the partial regression coefficient for the A step-wise regression was performed by taking all seven factor scores as the independent variables, and the dependent variable is one of the four success dimensions mentioned above. Those variables that significantly impacted the dependent variable would be considered as the critical success factor. At each level described above, the multiple correlation coefficient (R), the coefficient of determination, adjusted R 2 and coefficient B values of the model were computed. Those variables with significance value less than 0.05 would be recognized as candidates for being critical success factors. From the analysis done, we finally arrived at the list of the Most Significant Variables/ Success Factors as shown in Table 5.4. Table 5.4: Summary of the Outcome of the Regression Analysis Dependent Variable Independent Variable Quality Scope Time Cost COTS Software Development Techniques 0.505** 0.559** 0.322** 0.517** Healthy Organizational Environment 0.461** 0.345** 0.301** 0.306** High Caliber Team Capability Strong Customer Involvement 0.150* 0.232* Vendor Evaluation Strategy 0.292** 0.276** 0.391** Vendor s Reputation * Pricing Strategy 0.229* 0.264** 0.259* N R R Adjusted R F Probability > F *p<.05, **p<.01 Vikram Bali, Ph.D. Thesis

15 The results show that for success dimension of Quality, four success factors are considered significant, namely, COTS software development technique, healthy organizational environment, vendor evaluation strategy and strong customer involvement. For scope dimension, COTS software development technique, healthy organizational environment, vendor evaluation strategy and pricing strategy are considered significant success factors. For time dimension all the factors are significant except for high caliber team capability. Lastly, for the success of COTS project in terms of cost, the three significant factors are COTS software development technique, healthy organizational environment and pricing strategy. With the above observation, the results of hypotheses testing can be concluded as follows: out of 28 research hypotheses, a total of 17 hypotheses were supported while the remaining 11 were rejected. Those hypotheses were rejected due to their low coefficient values and high probability level for their corresponding null-hypotheses, that means, the presence of those factors did not make a significant contribution to the value of the success dimension. Table 5.5 summarizes the results of the hypotheses testing. The 17 supported hypotheses are highlighted and labeled with a check mark ( ). Those without the check mark are rejected hypotheses. Table 5.5: Summary of Hypotheses Testing Results Factor No. Independent Variable Quality Scope Time Cost 1. COTS Software Development Techniques H1a ( ) H1b ( ) H1c ( ) H1d ( ) 2. Healthy Organizational Environment H2a ( ) H2b ( ) H2c ( ) H2d ( ) 3. High Caliber Team Capability H3a H3b H3c H3d 4. Strong Customer Involvement H4a ( ) H4b H4c ( ) H4d 5. Vendor Evaluation Strategy H5a ( ) H5b ( ) H5c ( ) H5d 6. Vendor s Reputation H6a H6b H6c ( ) H6d 7. Pricing Strategy H7a H7b ( ) H7c ( ) H7d ( ) Vikram Bali, Ph.D. Thesis

16 5.4 ANALYSIS OF THE RESEARCH QUESTIONS Based on the outcome of the regression analysis and the hypotheses testing results above, we now are in the position to answer the research questions posed at the beginning of the study. Although the success level was calculated for each of the four success dimensions as opposed to the overall project success level, with each dimension carrying a distinct aspect to the perception of success, the frequency of supported hypotheses can be used to generalize the overall perception of success. The answers for each of the research questions are discussed in each sub-section below. Research Question 1: Are these seven factors truly the critical success factors of COTS based software projects? From the findings we can conclude that, out of seven factors, six have supported the hypotheses. The factors considered as the candidates of critical success factors are mentioned as follows: Factor 1 : COTS Software Development Techniques (in terms of Quality, Scope, Time and Cost) Factor 2 : Healthy Organizational Environment (in terms of Quality, Scope, Time and Cost) Factor 4 : Strong Customer Involvement (in terms of Quality and Time) Factor 5 : Vendor Evaluation Strategy (in terms of Quality, Scope and Time) Factor 6 : Vendor s Reputation (in terms of Time) Factor 7 : Pricing Strategy (in terms of Scope, Time and Cost) Factor 3, i.e. high caliber team capability, is not supported by any of the hypotheses. Previous studies also supported that technical, vendor selection and pricing are the key factors for successful COTS implementation projects. Vikram Bali, Ph.D. Thesis

17 Research Question 2: What is the relative importance of each factor when compared to other factors? Based on the hypotheses testing result, we can see that COTS software development technique and Organizational Environment had the most hypotheses supported (all four) followed by vendors evaluation strategy and pricing strategy (three each), followed by Strong customer involvement (two) and finally vendor s reputation (one). However, none of the hypotheses were supported for high caliber team capability. Based on the frequency, critical success factors are ranked and are given in Table 5.6. Table 5.6: Ranking of Critical Success Factors Rank Factor Hypotheses Supported Selection in the Model Frequency Value COTS Software Development Techniques H1a, H1b, H1c, H1d Healthy Organizational Environment H2a, H2b, H2c, H2d Vendor Evaluation Strategy H5a, H5b, H5c Pricing Strategy H7b, H7c, H7d Strong Customer Involvement H4a, H4c Vendor's Reputation H6a Vikram Bali, Ph.D. Thesis

18 Research Question 3: Is there a difference among those five factor categories in terms of their impact on the success of COTS based software project. Based on the regression analysis and hypotheses testing results, it was found that there was a significant difference among those five factor categories, namely, Technical, Organizational, People, Vendor Selection, and Cost. It was evident that the Technical Factor which included COTS software development technique and Organizational Factor which included organizational environment were the most influential in impacting the success of COTS based software projects, as they covered all the four success dimensions. This was followed by Vendor Selection Factor (which includes vendor s evaluation strategy and vendor s reputation) and Cost Factor (which includes pricing strategy) as these factors covered three success dimensions. Finally, People Factor in terms of strong customer involvement impacted the success in terms of Quality and time, but high caliber team capability under people factor failed to make impact on any of the success dimensions. Table 5.7 summarizes the level of impact of the five factor categories on the perceived success of COTS based software projects based on the above discussion. Table 5.7: Impact Level of Factor Categories on Project Success Rank Factor Dimension Hypotheses Supported Success Dimension 1 Technical H1a, H1b, H1c, H1d Quality, Scope, Time, Cost 2 Organizational H2a, H2b, H2c, H2d Quality, Scope, Time, Cost 3 Vendor Selection H5a, H5b, H5c, H6c Quality, Scope, Time 4 Cost H7b, H7c, H7d Scope, Time, Cost 5 People H4a, H4c Quality, Time 5.5 PRACTICAL IMPLICATIONS INVOLVED IN COTS BASED SOFTWARE DEVELOPMENT PROJECTS This study carries implications for the software professionals, involved in COTS based software development projects. First, the study reveals that the Technical, Organizational and Vendor Selection factors are important and lead to the success of Vikram Bali, Ph.D. Thesis

19 COTS based software projects. As per this research, the Technical Factor dimension includes the COTS software development techniques that deal with the issue of compatibility, integration, adaptability and open to work with other systems. As COTS based software systems are developed by integrating various COTS components, it becomes very important to handle these technical issues during the design phase of software development. If these issues are handled carefully, the probability of the success of the software project, in terms of quality, scope, time and cost, will also be high. Secondly, healthy organizational environment measured in terms of well-defined organizational structure, strong management support and clearly defined goals and objectives is also important factor dimension for the success of software project in all the four success dimensions. Thirdly, the study also suggests that the COTS vendor selection is an important factor for the success of software project in terms of quality, scope and time. As COTS based software are developed by integrating various COTS components which are purchased from different vendors from the software market. Therefore, this study highlights that while purchasing the COTS software product from the market the software professionals must evaluate the COTS vendors on the basis of their reputation, delivery time, after sales support and vendor s support infrastructure. Many of the researchers in the past have done a lot of work in the area of COTS evaluation and selection. One of the interesting findings of the study is that the vendor s reputation has a negative impact on success dimension of time. If the vendor s reputation is not good, the software project would not be completed on time. If the vendor does not deliver the COTS components on time then the development of the software would get affected or in other words delayed. Last but not least, the study also suggests that the software professionals must consider the price, license fee and maintenance fee as important parameters while purchasing the COTS software products from the vendors in the software market. Vikram Bali, Ph.D. Thesis