An empirical analysis of risk components and performance on software projects

Transcription

1 The Journal of Systems and Software 80 (2007) An empirical analysis of risk components and on software projects Wen-Ming Han, Sun-Jen Huang * Department of Information Management, National Taiwan University of Science and Technology, 43, Sec. 4, Keelung Road, Taipei, Taiwan Received 1 October 2005; received in revised form 18 April 2006; accepted 29 April 2006 Available online 14 June 2006 Abstract Risk management and enhancement have always been the focus of software project management studies. The present paper shows the findings from an empirical study based on 115 software projects on analyzing the probability of occurrence and impact of the six dimensions comprising 27 software risks on project. The MANOVA analysis revealed that the probability of occurrence and composite impact have significant differences on six risk dimensions. Moreover, it indicated that no association between the probability of occurrence and composite impact among the six risk dimensions exists and hence, it is a crucial consideration for project managers when deciding the suitable risk management strategy. A pattern analysis of risks across high, medium, and low- software projects also showed that (1) the requirement risk dimension is the primary area among the six risk dimensions regardless of whether the project belongs to high, medium, or low; (2) for medium- software projects, project managers, aside from giving importance to requirement risk, must also continually monitor and control the planning and control and the project complexity risks so that the project can be improved; and, (3) improper management of the team, requirement, and planning and control risks are the primary factors contributing to a low- project. Ó 2006 Elsevier Inc. All rights reserved. Keywords: Software project management; Software risk management; Risk exposure; Project 1. Introduction Software development is a highly complex and unpredictable activity. The Standish Group CHAOS Report in 2004 indicated that 53% of software projects were unable to deliver on schedule, within budget, and with the required functions, while 18% of the software projects were cancelled (Standish Group International, 2004). This stresses the fact that software projects pose various risks and daunting tasks for many organizations (Charette, 2005; Chris and Christine, 2003; Hoffman, 1999; McConnell, 1996). Now, as organizations invest substantial resources and effort on software development, controlling the risks associated with software projects becomes crucial (Kumar, * Corresponding author. Tel.: ; fax: address: huang@cs.ntust.edu.tw (S.-J. Huang). 2002). Hence, understanding the nature of the various software risks and their relationship with project has become increasingly important since the risk management strategy and plan depends on it. Developing an efficient and suitable risk management strategy depends on understanding two basic components probability of occurrence and impact on project (Boehm, 1991). The probability of occurrence of each software risk is different and its degree of impact on project cost, schedule and quality is also different. Hence, these two software risk components must be taken into consideration to develop a good software risk management strategy and plan. If not, the real benefits of the risk management activity may be lower than the resources invested (Boehm, 1991; Longstaff et al., 2000; Kumar, 2002; DeMarco and Lister, 2003). Although several articles have already identified various software risks, we currently lack an understanding of the /$ - see front matter Ó 2006 Elsevier Inc. All rights reserved. doi: /j.jss

2 W.-M. Han, S.-J. Huang / The Journal of Systems and Software 80 (2007) relative probability of occurrence and the various impacts of different software risks across projects in general. If project managers do not realize the natures of the different kinds of software risks and their relationship to project, they cannot develop an effective and appropriate strategy to mitigate or control them. Many studies have proven that proper management of software risks affects the success of software development projects (Jiang and Klein, 2000; Wallace and Keil, 2004). However, previous studies on software risk management failed to analyze the gap or pattern between software risks and project. For example, we can identify the software risks that negatively affect the project and thus, should be well-controlled in order to improve the project. Such insights regarding the patterns between software risks and project can assist project managers in developing a software risk management strategy that can mitigate the risks associated with software development projects. The present paper explores the relationships between software risks and their impact on project. The term project is defined as the degree to which the software project achieves success in the perspective of process and product (Nidumolu, 1996). In the present study, we began to address this issue by analyzing the probability of occurrence and impact on project of 27 software risks. These risks are classified into six dimensions namely, user, requirement, project complexity, planning and control, team, and organizational environment. A dataset from 115 historical software projects were analyzed using the MANOVA and Two-Step Cluster Analysis methods. These analyses addressed the following questions: (1) What are the different patterns for both the probability of occurrence and the impact in six software risk dimensions? (2) What relationships exist among the software risks in the three project clusters (high, medium, and low)? (3) What are the top 10 software risks and what are their associated probability of occurrence and impact? risk profile for a software project. Boehm (1991) surveyed several experienced project managers and developed a top 10 risk identification checklist for TRW. Barki et al. (1993) conducted a comprehensive review of software risk related studies from 120 projects in 75 organizations, then proposed 35 measures for software risks which were further classified into five dimensions: technological newness, application size, expertise, application complexity and organizational environment. Sumner (2000) used a structured interview to compare the differences between software risks within MIS and ERP projects, and proposed nine risks that were unique in ERP projects. Longstaff et al. (2000) proposed a framework named Hierarchically Holographic Modeling (HHM) and identified seven visions in systems integration that included 32 risks. Kliem (2001) identified 38 risks in BPR projects which were classified into four categories: people, management, business, and technical Addison (2003) used a Delphi technique to collect the opinion of 32 experts and proposed 28 risks for E-commerce projects. Meanwhile, Carney et al. (2003) designed a tool called COTS Usage Risk Evaluation (CURE) to predict the risk areas of COTS products in which he identified four categories comprising 21 risk areas. A summary of the software risks identified in the literature is shown in Table 1. Wallace et al. s work (2004) defined 27 software risks which were classified into six dimensions as shown in Table 2. This structure was utilized in this study to investigate the effects of risks on project Risk assessment method A risk assessment method is used to quantify the degree of importance of software risks to project. The importance of software risks usually is expressed as both the probability of occurrence and the impact on project. Three well-known software risk assessment methods in the literature are SRE, SERIM and DoD Guide. 2. Background This section discusses the various risks affecting the of software projects. Next, three of the wellknown risk assessment methods in the literature are introduced and compared. The Department of Defense (DoD) Risk Management Guide that was adopted to perform this empirical study is illustrated in this section Software risks Software risks are multifaceted and thus difficult to measure even though several studies have already been conducted since McFarlan (1981) identified three dimensions of software risks namely, project size, technology experience, and project structure. He suggested that project managers need to develop an aggregated software Table 1 Summary of risks in software development Author (year) Research area Dimensions Software risks McFarlan (1981) Common 3 54 Boehm (1991) Common 0 10 Barki et al. (1993) Common 5 35 Sumner (2000) ERP 6 19 Longstaff et al. (2000) Systems integration 7 32 Cule et al. (2000) Common 4 55 Kliem (2001) BPR 4 38 Schmidt et al. (2001) Common Houston et al. (2001) Common 0 29 Murthi (2002) Common 0 12 Addison (2003) E-commerce Carney et al. (2003) COTS 4 21 Wallace et al. (2004) Common 6 27

3 44 W.-M. Han, S.-J. Huang / The Journal of Systems and Software 80 (2007) Table 2 Software risks considered in this study Risk dimension Abbreviation Software risk User User1 Users resistant to change User2 Conflict between users User3 Users with negative attitudes toward the project User4 Users not committed to the project User5 Lack of cooperation from users Requirement Reqm1 Continually changing system requirements Reqm2 System requirements not adequately identified Reqm3 Unclear system requirements Reqm4 Incorrect system requirements Project complexity Comp1 Project involved the use of new technology Comp2 High level of technical complexity Comp3 Immature technology Comp4 Project involves use of technology that has not been used in prior projects Planning & Control P & C1 Lack of an effective project management methodology P & C2 Project progress not monitored closely enough P & C3 Inadequate estimation of required resources P & C4 Poor project planning P & C5 Project milestones not clearly defined P & C6 Inexperienced project manager P & C7 Ineffective communication Team Team1 Inexperienced team members Team2 Inadequately trained development team members Team3 Team members lack specialized skills required by the project Organizational environment Org1 Change in organizational management during the project Org2 Corporate politics with negative effect on project Org3 Unstable organizational environment Org4 Organization undergoing restructuring during the project Table 3 Various impacts of the risk assessment Impact SRE SERIM DoD Guide Cost X X X Schedule X X X Technical X X X Team X Support X The Software Risk Evaluation (SRE) method was developed by the Software Engineering Institute (SEI) to identify, analyze, and develop mitigation strategies for controlling risks (Williams et al., 1999). Version 2.0 of the SRE Method, released in 1999, describes the rating of the probability of occurrence on a scale of one to three, while the impact was defined by four components: cost, schedule, support and technical. The SRE impact components, however, did not consider the team issue which in turn, has become a critical factor in modern software projects (Jiang et al., 2000). Karolak (1997) proposed a method called Software Engineering Risk Management (SERIM) to predict risks and to provide corrective action in the software development phase. The SERIM identified 10 software risks and came up with 81 related questions that contained metrics to measure the software risks. The relationship between software risks were identified using three of the risk elements: cost, schedule and technical. However, as in the case of the SRE Method, SERIM did not include the team issue. The Risk Management Guide for DOD Acquisition (2003), released by the Secretary of Defense and the Defense Acquisition University (DAU), proposed a risk process model for how an organization identifies, assesses, and manages risks during software development. In this method, five levels were identified to assess the probability of occurrence of software risks namely, Remote, Unlikely, Likely, Highly likely and Near certainty. The impact of software risks represents the degree of influence on project. The types of impact of software risks on project in the DoD Guide includes technical, cost, schedule and team. A comparison of the various impacts for the above-mentioned software risk assessment methods is shown in Table 3. Since the DoD method contains diverse types of impact as well as the clear assessment criteria and description for each probability scale of occurrence and impacts, the method was adopted in the present study. Hence, the four types of impact defined by the DoD Risk Management Guide were adopted in this study. The assessment of the probability of occurrence and impact of software risks based on the DoD Risk Management Guide is shown in Table Data collection This study is based on data collected via a web-based survey. To increase the friendliness and usability of the survey, the 11 design principles proposed by Dillman et al. (2002) were adopted. For example, Use a scrolling design that allows respondents to see all questions unless skip patterns are important is one of them. The survey contained four sections. The first section explained the purpose of this study and encouraged project managers to participate in it. The second section requested the respondents to provide general information on their most recently completed software project. The third section listed 27 software risks and asked the respondents to assess the probability of occurrence and impact for each risk according to the DoD Risk Assessment Method in Table 4. Finally, the fourth section listed seven measures used to evaluate the project. There are organized into process

4 W.-M. Han, S.-J. Huang / The Journal of Systems and Software 80 (2007) Table 4 DoD risk assessment method Level Probability of Impact occurrence Technical Cost Schedule Team 1 Remote Minimal or no impact Minimal or Minimal or no impact None no impact 2 Unlikely Acceptable with some reduction <5% Additional resources required. Able to meet Some impact in margin need dates 3 Likely Acceptable with significant 5 7% Minor slip in key milestone. Not able to meet Moderate impact reduction in margin need dates 4 Highly likely Acceptable; no remaining margin 7 10% Major slip in key milestone or critical path impacted Major impact 5 Near certainty Unacceptable >10% Cannot achieve key team or major program milestone Unacceptable and product dimensions (Nidumolu, 1996; Rai and Al-Hindi, 2000). Due to space limitations, the whole questionnaire could not be included in the present paper. However, selected parts of the questionnaire are presented in Appendix A. The whole questionnaire can be made available upon request. A pretest was conducted to verify content validity and readability through personal interviews with five domain experts from academia and the software industry. Based on their feedback, the initial questionnaire was modified to improve the wording. A total of 300 project managers were invited and 135 surveys were returned within a span of two weeks. After checking the data received, 20 incomplete questionnaires were discarded. As a result, only 115 became valid survey questionnaires yielding a response rate of 38.33%. The number of years of work experience of the respondents ranged from 1 to 29 years, and the average was 10 years. To test the potential bias of the responses, the t-test method was used to compare the demographics of the early respondents versus the late ones (Armstrong and Overton, 1997). No significant difference was found (p = 0.983). Thus, the result indicated that the collected software projects can be combined for further analysis. Table 5 summarizes the 115 collected software projects. 4. Data analysis and results 4.1. The relationship of the probability of occurrence and impact in six risk dimensions It is important for project managers to explore patterns in the probability of occurrence and impact among risk dimensions. For example, do different risk dimensions Table 5 Profile of the collected 115 software projects Project attribute Mean Standard deviation Minimum Maximum Project duration (months) Delay time (percentage) Cost overlay (percentage) Staff turnover (percentage) N = 115 projects Table 6 The probability of occurrence and composite impact among risk dimensions Risk dimension Probability of occurrence Composite impact Risk exposure User Requirement Project complexity Planning and control Team Organization environment Baseline threshold show similar probability of occurrence? If not, what risk dimensions have a higher probability of occurrence? The MANOVA method is a widely-used statistical technique for verifying if the samples from two or more populations have equal means. This method was adopted to test whether or not the means of the risk components (probability of occurrence and composite impact) of the six risk dimensions are statistically different. The value of the composite impact of software risks was computed by averaging the impacts of software risks on the technical, cost, schedule and team. The verification for the assumptions of the MANOVA method (i.e., normality and homogeneity) was also performed and no violation of the assumptions was found in this empirical study. The results of the MANOVA indicated that the probability of occurrence and composite impact of the software risks of the six risk dimensions were significantly different. 1 The means of the probability of occurrence and composite impact among the six risk dimensions are shown in Table 6. A higher mean for a risk dimension indicates a higher possibility of occurrence or degree of influence. The baseline threshold, which was computed by averaging the six risk dimensions for each risk component, is a measure of the comparison criterion. Risk exposure was computed by multiplying the probability of occurrence with the composite impact (Boehm, 1991). Project managers are often asked to indicate what type of software risks frequently occur. As shown in Table 6, the baseline threshold of the probability of occurrence is 1 The MANOVA analysis results can be provided upon request.

5 46 W.-M. Han, S.-J. Huang / The Journal of Systems and Software 80 (2007) Only the two risk dimensions of requirement and project complexity exceeded the baseline threshold. This indicates that these two dimensions of software risks occur more frequently than the others. They are followed by the dimensions of team, planning and control and user risk dimensions. The organizational environment risk dimension does not occur as often as the other dimensions. In contrast to the probability of occurrence of software risks, the information about composite impact can assist the project managers in understanding the degree of negative effects on the project when these risks occur. When the composite impact of each risk dimension was compared with the baseline threshold (2.42) in Table 6, the requirement and planning and control risk dimensions showed a higher impact on the project. The risk dimensions of team, organizational environment, project complexity and user posed a lesser degree of impact on the project. Further analysis of Table 6 reveals that the orderings of the probability of occurrence and composite impact among the six risk dimensions were inconsistent. To confirm this point, Kendall s W rank test was used to verify whether these two orderings were significantly different. The result (p = 0.191) indicated that there is no significant association between the probability of occurrence and composite impact among the six risk dimensions. Risk exposure represents the overall importance of software risks and thus, helps the project managers to prioritize them and to develop an appropriate risk mitigation plan accordingly. Table 6 indicates that the requirement risk dimension has the highest probability of occurrence and composite impact and hence, is the principal source of software risks. The planning and control risk dimension has the second highest composite impact but has a lower probability of occurrence. In third place are the project complexity and team risk dimensions, both of which have the same value of risk exposure. However, the degrees of their probability of occurrence and composite impact are different. The Project complexity risk dimension has a higher probability of occurrence but a lesser composite impact as compared to the team risk dimension. This implies that the project risk managers need to employ different risk mitigation plans or strategies for these two risk dimensions accordingly. The remaining risk dimensions user and organizational environment exhibit even lesser probability of occurrence and composite impact. An organization or project manager cannot avoid all software risks since the resources of an organization or a project are limited. Hence, project managers must concentrate on those software risks with a higher risk exposure, and adopt a cost effective risk management strategy. An understanding of the nature of software risks the probability of occurrence and composite impact on project, can assist the project managers in adopting appropriate risk mitigation strategies for each of the different dimensions of software risk. Take the Project complexity risk dimension as an example. It has a higher probability of occurrence but a lower composite impact. Hence, it would be better for the project managers to adopt a strategy that reduces its probability of occurrence rather than lowers its degree of impact on the project The relationship between risk dimensions and impact The second topic of the present investigation focuses on examining the relationship between each risk dimension and its impact on the project. This addresses the question of whether or not there is a difference among the four types of impact caused by each risk dimension. Such information can assist the project managers in creating an appropriate risk management strategy. The MAN- OVA method was used to analyze the differences in the degree of impact of each risk dimension. The results show that the five risk dimensions have statistically significant differences (p < 0.05) with the four types of impact, except for the organizational environment (p = 0.550). The probable reason is that the concept of organizational environment is more abstract and broader and hence, the project managers had difficulty in distinguishing the degree of differences from the various impacts. Table 7 shows that the degrees of influence of requirement and planning and control risk dimensions are above the baseline threshold for all of the various impacts, while the rest are all below the baseline threshold. This reveals that if the project managers do not employ the appropriate risk management strategy for each of the risk dimensions, the efficiency or possible benefits from the software risk management will be negatively affected. Table 8 depicts the detailed information regarding the probability of occurrence and impact among the 27 software risks. This can be used as baselines to assist an organization in triggering risk-handling activities and to understand the strengths and weaknesses of its software development capability Patterns in risk dimensions across the levels of project The third topic of the present investigation concerns the relationship between risk dimension and project perfor- Table 7 The composite impact of four types among risk dimensions Risk dimension Technical Cost Schedule Team User Requirement Project complexity Planning and control Team Organization environment Baseline threshold

6 W.-M. Han, S.-J. Huang / The Journal of Systems and Software 80 (2007) Table 8 The probability of occurrence and impact among software risks Software risk Probability of occurrence Impact Technical Cost Schedule Team User risk dimension Users1 a Users Users Users Users Requirement risk dimension Reqm Reqm Reqm Reqm Project complexity risk dimension Comp Comp Comp Comp Planning & Control risk dimension P & C P & C P & C P & C P & C P & C P & C Team risk dimension Team Team Team Organization environment risk dimension Org Org Org Org a Please refer to Table 2 for the meaning of the abbreviations in the column of this table. mance. Project is determined by averaging the five product measures and two process measures. In order to have an objective classification of the levels of project, the Two-Step Cluster Analysis, a tree-based technique in SPSS package, was used (SPSS, 2003) in this study. The optimal number of clusters was decided through Schwarz s Bayesian Criterion. It resulted in three clusters which were divided into low (n = 16), medium (n = 45) and high (n = 54) project as shown in Table 9. The values in Table 9 represent the risk exposure of a specific project cluster by averaging the risk exposures of all the projects within this cluster. Several interesting findings could be observed from two points of view namely, the pattern of the software risk dimensions and the individual project cluster. For the pattern of the software risk dimensions, the radar chart, as shown in Fig. 1, provides a clear graphical Table 9 Cluster means for the six risk dimensions Risk dimension High (n = 54) Medium (n = 45) Low (n = 16) User Requirement Project complexity Planning and control Team Organization environment Baseline threshold Project Organization Environment Team User Planning & Control Requirement Project Complexity Low Medium High Fig. 1. Risk radar chart. description showing that all risk dimensions obviously increased from the low to high- clusters. This finding provides empirical evidence that software risks decrease the degree of project (Jiang and Klein, 2000; Wallace and Keil, 2004). The area enclosed in the radar chart represents the risk exposure in six risk dimensions for one of three software project clusters. The radar chart shows that the difference between the regions of low- and medium- software projects is much larger than the difference between the medium- and the high- software projects. This suggests that there is an inverse and nonlinear relation between software risks and project. In the high- cluster of the software projects, the requirement risk dimension is the primary factor for lowering the project since the values of the other five risk dimensions do not exceed the baseline threshold (5.34). The result suggests that managing the requirement risks is critical to achieving the best project because even the high- projects still have a high degree of the requirement risk dimension. For the medium- project, the requirement, project complexity and planning and control risk

7 48 W.-M. Han, S.-J. Huang / The Journal of Systems and Software 80 (2007) Table 10 The top 10 software risks Rank Software risk Probability of occurrence Technical Cost Schedule Team Risk exposure 1 Continually changing system requirements (Reqm1) System requirements not adequately identified (Reqm2) Unclear system requirements (Reqm3) Lack of an effective project management methodology (P & C1) Incorrect system requirements (Reqm4) Poor project planning (P & C4) Inadequate estimation of required resources (P & C3) Project involved the use of new technology (Comp1) Project progress not monitored closely enough (P & C2) Corporate politics with negative effect on project (Org2) dimensions are the factors that decrease the project because all of their values are above the baseline threshold (6.51). Hence, controlling these risks will help to improve the project. For the low- project, the major risk dimensions are the requirement, planning and control, and team because all their values are above the baseline threshold (9.22). Thus, continually monitoring these risk dimensions is a basic requirement for an effective software risk management. Further analysis of the data in Table 9 yields four important findings. Firstly, the requirement risk dimension of all the three project clusters exceeds the baseline threshold. This means that successfully managing the requirement risk is a basic requirement for achieving the desired software project. Secondly, the project complexity risk dimension exceeds the baseline threshold only for the medium- projects. Other clusters, however, do not have the same situation. Thirdly, only the planning and control risk dimension in the high- project does not exceed the baseline threshold. This suggests that the medium and low- projects need to carefully manage the planning and control risk so that they can have chances of improving their project. Lastly, only the team risk dimension in the low- project exceeds the baseline threshold. This suggests that not addressing well the risks associated with the development team can negatively affect project A list of the top 10 risks To determine the top 10 risks which lead to failure in software projects, the risk exposure of all the risks within the six dimensions were computed by multiplying the value of probability of occurrence with the various degrees of composite impact. Table 10 presents a list of the top 10 software risks and their associated probability of occurrence, degrees of the four various impacts, and risk exposure. The past studies (Boehm, 1991; Mursu et al., 2003; Wallace and Keil, 2004) emphasized the top 10 risks by simply listing them. The present study not only presents the top 10 software risks but also includes the information about their probability of occurrence and degree of impact. In this way, the project managers can have a better idea of how to manage these software risks. 5. Conclusion Achieving effective software risk management requires project managers to understand the nature of software risks. Thus, information about the probability of occurrence and impact of software risks on project can help the project managers to develop a better risk management strategy. This empirical study considered risk information on 115 software projects. The results indicate that a positive correlation does not exist between the probability of occurrence and impact among the six risk dimensions. The relationship between software risks and project was also examined in the high, medium, and low- projects. The results show that the requirement risk dimension is the principal factor affecting the project. Aside from this, one of the ways to improve project is by properly planning the development activities and reducing the project complexity. Likewise, if the project manager is unable to effectively manage the requirements of the whole project life cycle, and does not well-plan nor monitor the software risk management plan, the software projects is likely to perform poorly. The of the software projects could also benefit from exploring the relationship between risk components and some important attributes of software projects such as the type of software systems and project duration. These are the possible research topics for the future. Appendix A Selected parts of the questionnaire To what degree do you believe the probability of occurrence and four impacts of Team risks exist in your most recently completed software project? Please circle the response that best represents your judgment on the following scales based on the DoD Risk Management Assessment Method which is provided in the attached sheet.

8 W.-M. Han, S.-J. Huang / The Journal of Systems and Software 80 (2007) Inexperienced team members (Team1) Probability of occurrence Remote Near certainty Technical Minimal or no impact Unacceptable Cost Minimal or no impact >10% Schedule Minimal or no impact Cannot achieve key team or major program milestone Team None Unacceptable Inadequately trained development team members (Team2) Probability of occurrence Remote Near certainty Technical Minimal or no impact Unacceptable Cost Minimal or no impact >10% Schedule Minimal or no impact Cannot achieve key team or major program milestone Team None Unacceptable Team members lack specialized skills required by the project (Team3) Probability of occurrence Remote Near certainty Technical Minimal or no impact Unacceptable Cost Minimal or no impact >10% Schedule Minimal or no impact Cannot achieve key team or major program milestone Team None Unacceptable To what degree do you believe the product and process measures exist in your most recently completed software project? Please circle the response that best represents your judgment on the following scales. Strongly Strongly agree disagree The application developed is reliable The application is easy to maintain The users perceive that the system meets intended functional requirements The system meets user expectations with respect to response time The overall quality of the developed application is high The system is completed within budget The system is completed within schedule References Addison, T., E-commerce project development risks: evidence from a Delphi survey. International Journal of Information Management 23 (1), Armstrong, J., Overton, T., Estimating non-response bias in mail survey. Journal of Marketing Research 15, Barki, H., Rivard, S., Talbot, J., Toward an assessment of software development risk. Journal of Management Information Systems 10 (2), Boehm, B., Software risk management: principles and practices. IEEE Software 8 (1), Carney, D., Morris, E., Patrick, R., Identifying commercial off-theshelf (COTS) product risks: the COTS usage risk evaluation. Technical Report, CMU/SEI-2003-TR-023. Charette, R., Why software fails. IEEE Spectrum 42 (9), Chris, S., Christine, C., The State of IT Project Management in the UK University of Oxford, Templeton College. Cule, P. et al., Strategies for heading off IS project failure. Information Systems Management 17 (2), DeMarco, T., Lister, T., Waltzing with Bears: Managing Risk on Software Projects. Dorset House Publishing Company. Dillman, D. et al., Influence of plain vs. fancy design on response rates for web surveys. Joint Statistical Meetings. Available from: < Hoffman, T., % of IT departments fail to meet business needs. Computerworld 33 (41), 24. Houston, D., Mackulak, G., Collofello, J., Stochastic simulation of risk factor potential effects for software development risk management. Journal of Systems and Software 59 (3), Jiang, J., Klein, G., Software development risks to project effectiveness. The Journal of Systems and Software 52, Jiang, J., Klein, G., Means, T., Project risk impact on software development team. Project Management Journal 31 (4), Karolak, D., Software Engineering Risk Management. IEEE Computer Society. Kliem, R., Risk management for business process reengineering projects. Information Systems Management 17 (4), Kumar, R., Managing risks in IT projects: an options perspective. Information and Management 40, Longstaff, T. et al., Are we forgetting the risks of information technology? IEEE Computer 33 (12), McConnell, S., Rapid Development: Taming Wild Software Schedules. Microsoft Press.

9 50 W.-M. Han, S.-J. Huang / The Journal of Systems and Software 80 (2007) McFarlan, F., Portfolio approach to information systems. Harvard Business Review 59 (5), Mursu, A. et al., Identifying software project risks in Nigeria: an international comparative study. European Journal of Information Systems 12 (3), Murthi, S., Preventive risk management for software projects. IT Professional 4 (5), Nidumolu, S.R., Standardization, requirements uncertainty and software project. Information and Management 31 (3), Rai, A., Al-Hindi, H., The effects of development process modeling and task uncertainty on development quality. Information and Management 37 (6), Schmidt, R. et al., Identifying software project risks: an international Delphi study. Journal of Management Information Systems 17 (4), SPSS Inc., SPSS 12.0 User Guide and Tutorial. Standish Group International, Inc., Third Quarter Research Report. Sumner, M., Risk factors in enterprise-wide/erp projects. Journal of Information Technology 15 (4), US Department of Defense, Risk management guide for DOD acquisition fifth edition. Defense Acquisition University, Defense Systems Management College. Wallace, L., Keil, M., Software project risks and their effect on outcomes. Communications of the ACM 47 (4), Wallace, L., Keil, M., Arun, R., How software project risk affects project : an investigation of the dimensions of risk and an exploratory model. Decision Sciences 35 (2), Williams, R. et al., SRE Method Description & SRE Team Members Notebook Version 2.0. Software Engineering Institute, Carnegie Mellon University CMU/SEI-99-TR-029.