A Semi-Structured Process for ERP Systems Evaluation: Applying Analytic Network Process

A Semi-Structured Process for ERP Systems Evaluation: Applying Analytic Network Process Huan-Jyh Shyur shyur@mail.im.tku.edu.tw Department of Information Management Tamkang University Abstract This paper illustrates a four-step semi-structured process for ERP system evaluation. We suggest using the Analytic Network Process (ANP) for qualitative reviews of ERP, reviews involving multiple criteria and interdependency of properties. The ANP method is based on the feedback system framework of the well-known Analytic Hierarchy Process. A case study indicates that the evaluated aspects of the method are feasible and the method improves the quality of ERP system selection compared with traditional approaches. Keywords: Enterprise Resource Planning, Analytic Hierarchy Process, Analytic Network Process, Weighted Score Method. Journal of e-business, Vol. 5, No. 1, 2003/3 33

A Semi-Structured Process for ERP Systems Evaluation @à}&æ å k$ Äd Í$ š!z Dz 5û shyur@mail.im.tku.edu.tw µ± ç müçí b dt ø_ Öû¼ í$ Äd Í$Dz¼ ÊDz 2@à}&æ å (Analytic Network Process, ANP) V vj²ö Ç,Ä Ä È>Tàú²µ5 à }&æ å uj}&µ å (Analytic Hierarchy Process, AHP) Ñ! 1 p œ Fê 7AíhxX ÊõWð 2ªêÛ à û Fqì5²µ 1 º }&æ å í@àuø_ªw/üõ?t $ ÊDz$ Äd Í$ví²µ¹ ÉœÈ: $ Äd }&µ å }&æ å l} 34

1 Introduction Today s Enterprise Resource Planning (ERP) systems offer a wide variety of capabilities to support traditional business process areas such as manufacturing, human resources, finance and supply chain management. These systems incorporate with new information technologies such as data warehouses, internet access, client/server or multi-tiers structure to speed decision-making, reduce costs and give managers control over the whole business. It is obviously that successful ERP implementation can gather enormous benefits for companies. However, Holland and Light (1999) point out it can be disastrous for organizations that fail to manager the implementation process. Implementing an ERP system induces to massive change that needs to be carefully managed to take advantage of an ERP solution. Critical issues are presented in many previous researches to ensure successful implementation, such as top management commitment, project schedule and plans, personnel, business vision, ERP strategy, consultants, vendors and so on (Bingi et al., 1999; Holland and Light, 1999; Langenwalter, 2000). Prior to the implementation, the system evaluation process must be executed to identify which ERP system is selected. The decision-making process is a critical point in the life cycle of an ERP system. As we know software packages are designed for different target industries and company size; none of them can fit all organizations equally well. ERP system selection deliberately creates the foundation for successful implementation and maximum ROI. Owing to the complexity of the ERP systems and abundance of alternatives, a systematic process of selection can be arduous and expensive. However, when compared to the cost of software, hardware, and risk of failure, the cost is relatively inexpensive. Moreover, the impact of a bad decision strongly influences long-term business success. With over 300 various ERP providers on the market, there are two critical questions. One is where a company can start their evaluation process and the other is what the whole process is. According to Kontio s observation in many organizations, the information systems selection process typically is not well defined, each project finds its own approach to it, often under schedule pressure, and there are no mechanisms to learn from previous selection cases (Kontio, 1996). The selection of an appropriate ERP solution is a semi-structured decision problem since only part of the problem can be handled by a definite or accepted procedure such as standard investment calculations. On the other hand the decision maker needs to judge and evaluate all relevant and intangible business impact aspects (Bernroider and Koch, 2000; Laudon and Laudon, 1998). Yet the selection of the right ERP system is often a non-trivial task and requires careful consideration of multiple criteria and careful balancing between business requirements, technical characteristics, and financial issues. Journal of e-business, Vol. 5, No. 1, 2003/3 35

A Semi-Structured Process for ERP Systems Evaluation In this paper we summarize the ERP system evaluation criteria and reconstruct a semi-structured process for ERP system selection. Specially, we utilize the Analytic Network Process (ANP) model to address the issue of how to generate priorities for decisions involving multiple criteria and general types of dependence of criteria on alternatives, and criteria on criteria in the ERP system selection process. A real case experiment has been done to evaluate the proposed method for rating each alternative against the selection criteria. 2 Multi-Criteria Information System Decision Making Model Reviewing the previous researches, several quantitative or qualitative methods have been proposed to help organizations make decision in project of information systems selection (Anadalingam and Olsson, 1989; Buss, 1983; Sanathanam and Schniederjans, 1993). The decision technique usually used to evaluate IS projects is based on scoring methods, cost/benefit analysis, risk analysis, and ranking etc. Lee and Kim (2000) argued that scoring and ranking techniques are intuitively simple but they are insufficient for dealing with project interdependence. Baker and Freeland (1975) addressed that cost/benefit and risk analysis methods are inadequate treatment of multiple, often interrelated, criteria. When decision makers select their ERP system, financial criteria are frequently major factors. However other less easy quantifiable criteria such as technological risk and system flexibility must be considered together. Hwang and Yoon (1981) called such a decision making involving multiple criteria as multi-criteria decision-making (MCDM). On a deep study of these identified criteria, we may find some of the criteria are conflicting; many are not well-defined and hardly measurable, which increases the complexity of the decision process. In this section, we explore some of the MCDM methods. 2.1 Scoring Method and Ratio Scale Method Weighted Scoring method (WSM) is commonly used approach in IS projects selection. In generally, WSM is used when it is necessary to make a complex evaluation based on a number of criteria of differing importance. This approach involves assigning a relative weighting to each desirable criterion based on relative importance and allocating a score to each alternative for each criterion. The weighted score for each alternative is calculated by the following formula and the one with 36

the highest score is selected: score a = n (weight j score a j ), j=1 where subscript represents an alternative, and n represents the total number of criteria. WSM is very effective in dealing with complicated requirements and can greatly assist in the selection of a preferred supplier when the weighted scores are widely distributed (Kontio, 1996). However, problems can arise with this method when there is little difference between the total scores for different alternatives, thus complicating the justification of a decision to prefer one alternative to the other. An alternative study identified the multi-criteria decision technique known as the Analytic Hierarch Process (AHP) to be more appropriate for solving complicated problems. AHP was proposed by Satty (1980) as a nonlinear framework method of solving socio-economic decision making problems and has found its widest applications in multi-criteria decision-making, including software engineering and software selection (Finnie et al., 1995; Hong and Nigam, 1981; Min, 1992). We can regard AHP as the ratio scale approach to decision-making. By using the AHP to model a problem, one needs a unidirectional hierarchic structure that consists of at least three levels, a goal, criteria and alternatives to represent the problem. All the elements in the three levels should be well-defined prior to develop the hierarchy. The hierarchy depicts the structure of the decision problem and forms the foundation of the comparisons that have to be made in the remaining process. Satty put forth that hierarchies are a nature way for humans to organize their view of the world (Satty, 1980). The pairwise comparisons are the input of AHP models that calculate the relative priority of each alternative. For each corresponding criteria, the decision makers perform a series of pairwise comparisons, where two alternatives are compared at a time. The pairwise comparisons are represented in a matrix form. Within the matrix, the scores are conducted for a pair and a reciprocal value is automatically assigned to the reverse comparison. To derive the relative priority of each alternative with respect to one particular criterion, AHP uses the eigenvalues and eigenvectors of the pairwise comparison matrix (Satty, 1980). Next, AHP determines the relative importance of each criterion. This is done by means of the same process that was used in calculating the relative priorities of the alternatives. Once the two levels of pairwise comparisons are completed, the relative priorities for an alternative are multiplied by the importance of the corresponding criteria and summed over all criteria to determine the overall priorities. Journal of e-business, Vol. 5, No. 1, 2003/3 37

A Semi-Structured Process for ERP Systems Evaluation Alidi (1996) concluded that AHP is easy to use, intuitiveness and consensus building. However, one may argue that the AHP model is initially perceived as difficult as its calculation model is more complex and it involves a high number of paired-comparison. The large volume of individual assessments in the AHP method is perhaps one of its main weaknesses since the repetitive assessments may cause fatigue in decision makers. 2.2 Analytic Network Process The Analytic network process (ANP), which is the extension of AHP, is developed by Satty as well to generate priorities for decisions without making assumptions about a unidirectional hierarchy relationship among decision levels (Satty, 1996, 1999). To take the place of a linear top-to-bottom form of strictly hierarchy, the ANP model provides a looser network structure makes possible the representation of any decision problem. The relative importance or strength of the impacts on a given element is measured on a ratio scale similar to AHP. The major difference between AHP and ANP is that ANP is capable of handling interdependence of higher-level elements from lower level elements and the independence of the elements within a level by obtaining the composite weights through the development of a supermatrix. The supermatrix is a partitioned matrix, where each submatrix is composed of a set of relationships between two components or clusters in a connection network structure. Satty (1996) explains the concept corresponding to the markov chain process. For our discussion, we use the matrix manipulation based on the concept of Satty and Takizawa s in place of Satty s suppermatrix (Satty and Takizawa, 1986), which is much easier to understand. 3 Application of ANP to ERP Systems Evaluation ERP system selection is a semi-structured decision process, which involves multiple objectives or criteria to determining the priority for each system. Illa et al. (2000) suggested an evaluation process to create the fundamental for successful ERP implementation and maximum ROI. Langenwalter (2000) presented a tenstep ERP system selection process as well, which composes of a series of complex review and decision. We referred to these literatures and proposed a four-step evaluation process in this paper. To illustrate the process and show how ANP can be used in ERP system selection, an application based on practical experience and implementation in a PCB industry is presented in this session. A decision team involved 3 managers and 5 key users were created to perform the decision making process. All of the three managers had previous experience in COTS software evaluation. 38

Step 1: Study strategy and business processes. The initial step plays an important role in the whole process, which identifies improvement opportunities and develops some general ideas for the to-be system. To carry out this process, we spent more than one month to gather enough information through interviews with users and managers, observation of current operation process, and analysis of the system s documentation. Once the information is collected, some fundamental decision questions can be answered. For example, whether the business has to acquire an ERP system, what manufacturing strategies will the company choose, and what kinds of new services will be offered? According to the understanding of current business processes, the strengths and weaknesses of these processes can be analyzed and the improvement opportunities can be identified. The required functionalities for expected ERP system then can be evolved. Step 2: Create screening criteria to conduct a market research and narrow the field to few serious candidates. In step 2, the screening criteria are created to conduct a market research initiative looking for suitable ERP systems and narrow the field, where screening criteria are the minimum requirements about candidate systems. To execute this step, decision makers have to obtain enough minimum information on each expected systems. Langenwalter (2000) suggested three major screening criteria can potentially be used to initially narrow the field - industry type, manufacturer size, and technical platform. In our practical example, we bring into budget limit and supporting languages as supplementary criteria. In this illustrative example, five ERP systems are selected into the candidate list that apparently fit the major criteria reasonably well. Step 3: Rank and select the final lists. The third phase is the first step that the evaluation team is forced to make real comparisons and it is composed of a series of complex review and decision but short of an explicit approach. The decision problem faced by the evaluation team is to determine the priority of each candidate and select the finalists. However, the further evaluation process is especially difficult since the candidate packages have high similarities. To become aware of strengths and weaknesses of the various candidates in each criterion, people may review the RFP, meet with the providers, visit the reference sites, and participate in the package demo (Langenwalter, 2000). We invited each of the system providers to give a detail presentation and a discussion session. Journal of e-business, Vol. 5, No. 1, 2003/3 39

A Semi-Structured Process for ERP Systems Evaluation The screening criteria used in the previous step often are not appropriate to act as a basis in the current phase. We found intangible criteria become more important. A number of ERP system selection criteria have been proposed in practitioner and academic literatures (Enzweiler, 1997; Glazer, 1999; Illa et al., 2000). These criteria involve both the requirements that a company may currently have and will also need when they implement the ERP system and those requirements that they currently do not need but will require when they implement and maintain the system. However, each company may have their own set of criteria. Kontio (1996) presents a selection criteria definition process that essentially decomposes a goal into a hierarchical criteria set and each branch in this hierarchy ends in an evaluation attribute. Following the same process, we determined seven potential ERP system evaluation criteria formally through meetings with engineers, managers and decision makers in a practical example. Table 1 presents the proposed criteria and their related attributes. The seven criteria may not include all of the decision factors in ERP system selection. However, they are the most meaningful measures in our case and have been stressed in many leading ERP articles and books. Within this illustrative example, we thought that there is an existence of interdependence relationship among the seven defined criteria. A group decisionmaking process constructed the relationship based on the following recognitions. Good integrated solution, one-stop service, education support, and technical support will reduce the technological risk and make implementation process easier. The higher charged price for implementation and maintenance may let supplier tend to place more emphasis on customer care. A system with higher integration level induces lower technological risk since less customer development is needed. In order to reduce technological risk, we need a more flexible system. Figures 1 represents the type of interdependency network. The single arrows imply a one-way relationship. For example, the arrow that leaves from SI and feeds into TR infers that the attributes of criterion SI influence criterion TR. Due to the characteristics of this case study presented here we explore the appropriateness of ANP to allow for the explicit consideration of interactions in the decision making process. The remaining procedure involves five sub-steps and shown as follow: 40

Table 1: Evaluation Criteria and Related Attributes Selection Criteria Cost (CO) Supplier s support (SS) Evaluation Attributes License fee, Modular pricing, Maintenance, Documentation, Consultant fee, Resource utilization, Conversion cost. Vendor responsiveness, Consulting, Hot line, Training, Technical support personnel, Continuing enhancement, Time sharing access, Warranty, Documentation, Financial stability, Local branch office, Third vendor support, Growth of customer base, Active R&D. Technological risk (TR) Nonrobust and incomplete ERP packages, Complex and undefined ERP-to-legacy-system interfaces, Middleware technology bugs, Poor custom code, and poor system performance, Software maturity, Hardware maturity. Closeness of fit to the company s business (FB) Easy of implementation (EI) Flexibility to easy change as the company s business changes (FC) System Integration (SI) Main Target, Included functionality Shorter implementation time, User friendliness, Multisite implementation Adaptability, Openness for customer development, Openness for working with other systems Internal connectivity, External connectivity Journal of e-business, Vol. 5, No. 1, 2003/3 41

A Semi-Structured Process for ERP Systems Evaluation Figure 1: Relationship among criteria Table 2: Criteria Pairwise Comparison Matrix CO SS EI FB FC TR SI Vector Weights CO 1 1/5 1/5 1/5 5 5 1/5 0.063 SS 5 1 3 1/2 4 5 2 0.227 EI 5 1/3 1 1/5 4 5 2 0.146 FB 5 2 5 1 5 6 6 0.37 FC 1/5 1/4 1/4 1/5 1 3 1/5 0.04 TR 1/5 1/5 1/5 1/6 1/3 1 1/5 0.027 SI 5 1/2 1/2 1/6 5 5 1 0.127 Step 3.1: Without assuming the interdependence among criteria, the decision makers were asked to evaluate all proposed criteria pairwise. They responded questions such as: which criteria should be emphasized more in an ERP system, and how much more? The responses were presented numerically and scaled on the basis of Satty s proposed 1-9 scale (Satty, 1980), where 1 represents indifference between the two criteria and 9 is extremely preferred of the criterion under consideration over the comparison criterion. Each pair of criteria was judged only once. In our example, a reciprocal value is automatically assigned to the reverse comparison. The final result is represented in a matrix form (see Table 2). According to Satty s concept (Satty, 1996), the local priority vector of criteria was calculated by using the eigenvalue and eigenvector of the matrix. The obtained eigenvector is like (CO, EI, SS, FB, FC, TR, SI) = (0.63, 2.26, 1.45, 3.67, 0.39, 0.26, 1.27). Then we normalized the vector by dividing each value by its column total. The results represent the related local priority of these criteria, which are displayed as the last column in Table 2. Within this example interdependence was not considered until next step. 42

Table 3: Dependence among the seven criteria Criteria CO SS EI FB FC TR SI CO 1 0.13 0 0 0 0 0 SS 0 0.87 0.28 0 0 0.16 0 EI 0 0 0.64 0 0 0 0 FB 0 0 0.08 1 0 0 0 FC 0 0 0 0 1 0.08 0 TR 0 0 0 0 0 0.64 0 SI 0 0 0 0 0 0.12 1 Step 3.2: In this step, we considered and analyzed the dependence among the selection criteria. The decision makers examined the impact of all the criteria on each by using pairwise comparisons as well. They answered questions such as: which criterion will influence criterion TR more: SI or FC? and how much more? Totally, seven pairwise comparison matrices were developed. The normalized eigenvectors for these matrices are calculated and shown as seven columns in Table 3, where zeros are assigned to the eigenvector weights of the criteria from which a given criterion is given. The data in Table 3 mean seven criteria s degree of relative impact for each seven criteria. For example, the CO s degree of relative impact for SS is 0.13. Step 3.3: The relative importance of each criterion was first determined. In addition, seven pairwise comparison matrices were developed for calculation of the relative priorities of the five alternatives with respect to each criterion. The relative priorities are calculated by applying similar computational rules as used in step 3.1. For each matrix the eigenvalue and eigenvector were calculated. The data in each vector were normalized to sum to one as well. The results are shown in Table 4, in which each column represents the relative priority for each alternative with respect to a particular criterion. Step 3.4: The relative importance of the criteria considering interdependence now can be obtained by synthesizing the results from Steps 1 and Steps 2 as Journal of e-business, Vol. 5, No. 1, 2003/3 43

A Semi-Structured Process for ERP Systems Evaluation Table 4: Relative priorities of the five alternatives with respect to each criterion Evaluation Criteria Alternatives CO SS EI FB FC TR SI A1 0.28 0.15 0.11 0.21 0.17 0.21 0.04 A2 0.07 0.27 0.17 0.18 0.21 0.16 0.35 A3 0.13 0.21 0.21 0.16 0.15 0.31 0.24 A4 0.25 0.23 0.26 0.12 0.24 0.23 0.16 A5 0.27 0.14 0.25 0.33 0.23 0.09 0.21 follows (1): W c = CO S S EI FB FC TR S I = 1 0.13 0 0 0 0 0 0 0.87 0.28 0 0 0.16 0 0 0 0.64 0 0 0 0 0 0 0.68 1 0 0 0 0 0 0 0 1 0.08 0 0 0 0 0 0 0.64 0 0 0 0 0 0 0.12 1 0.06 0.23 0.15 0.37 0.04 0.03 0.13 = 0.09 0.25 0.01 0.38 0.04 0.02 0.13 (1) According to what we calculated, FB, SS, and SI were three of the most important considering factors relating to ERP system evaluation in this company, which was confirmed by the decision makers. Step 3.5: Now, the overall priorities for the alternatives can be calculated by multiplying WA by WC, where the relative priorities for an alternative are multiplied by the importance of the corresponding criteria and summed over all criteria (2). W a W c =.28.15.11.21.17.21.04.07.27.17.18.21.16.35.13.21.21.16.15.31.24.25.23.26.12.24.23.16.27.14.25.33.23.09.21.09.25.01.38.04.02.13 = Our final results in the phase were (A1, A2, A3, A4, A5) = (0.170, 0.217, 0.189, 0.186, 0.248). For alternative A5, the overall weighted priority was 0.248, which was assigned the highest weight. For alternatives A2, A3, A4, and A1, 44.17.22.19.19.05 (2)

the overall weighted priorities are 0.217, 0.189, 0.186, and 0.17 respectively. The alternatives with the largest priority weight should be the one selected. It should be mentioned that an ANP based MCDM support system developed in Tamkang University was used for all of the above judgments and complex calculations. Step 4: Prioritize and bring additional insight to some of the intangibles to select the winner. Before the top manager chose the ERP system with the highest weighted priority, a refinement process was performed to confirm the results. The decision team reviewed and refined the application of the list of criteria to each final candidate system. Each provider was requested to demonstrate the system, to help the decision team to obtain a much deeper knowledge on its functionality and adaptability to the company. The decision team visited two reference sites of the winning software as well, to see the software in actual use in a company. In this case, alternative A5 was selected as the final winner after the refinement process. 4 Model Evaluation During the formal decision-making process, an experiment had been done simultaneously to evaluate the proposed model. We expected that the ANP model would provide decision makers more confidence in the decisions they made. Before the ANP model was conducted to the third phase decision-making, three managers in the decision team were invited to take a pretest using WSM method to evaluate each of the five alternatives independently. Prior the pretest, the managers spent more than 3 hours to discuss the importance of each proposed criteria. After that, they were asked to assign a relative weighted score between one and five to each criterion independently. Once the relative weighted scores were identified, the managers were requested to allocate an additional score between zero and five for each alternative in each criterion as well. According to the WSM method, three sets of overall weighed scores were calculated and represented in Table 5. It appears that managers 1 and 3 assigned the highest weighted score for system A5. However, manager 2 gave the highest weight score to system A2. In addition, systems A2 and A3 have close scores but the weighted priorities assigned by the three managers are inconsistent. One week later, to let the members in the decision team felt comfortable with the proposed method, they were requested to apply ANP model to conduct a simulated decision-making process based on their currently obtained information. Satty s proposed 1-9 scale was used in the score assigning process. Since there Journal of e-business, Vol. 5, No. 1, 2003/3 45

A Semi-Structured Process for ERP Systems Evaluation Table 5: Evaluation results using WSM Alternatives Managers A1 A2 A3 A4 A5 1 103 114 111 106 128 2 92 109 106 94 108 3 88 106 108 91 113 Table 6: Evaluation results using ANP Alternatives Managers A1 A2 A3 A4 A5 1 0.152 0.214 0.202 0.176 0.256 2 0.143 0.223 0.207 0.179 0.248 3 0.154 0.221 0.194 0.169 0.262 were seven criteria with interdependence, 33 pairwise comparisons were needed to evaluate their relative importance; and an additional 70 pairwise comparisons were required to evaluate the performance of each alternative on each criterion. In this experiment, three of the managers were asked to perform the process independently as well for the purpose of comparison. After the simulated decisionmaking process, all the members in the decision team felt to perform the process may cause them fatigue since the large number of individual assessments. However, the managers perceived that the evaluation process is easier compared with the WSM method especially working with our computer aided decision tool. The average time to make a pairwise comparison is less than 30 seconds in this case. As the ANP method produced ratio scale ranking, the surprising result show that the overall priorities for the alternatives identified by the three evaluators are the same (A5 > A2 > A3 > A4 > A1). The numbers in Table 6 are the normalized weighted scores for each alternative. A formal group decision meeting was conducted after the experiment. Using the graphical tool, all evaluators discussed and assigned the score for each paired comparison together. The overall duration of the assessment session was shorter than what we expected. They defined the normalized weighted scores for each alternative as (A1, A2, A3, A4, A5)=(0.163, 0.228, 0.193, 0.184, 0.242). Once again, the result presents a consistency with the previous outcome from the individual evaluation process. 46

5 Conclusion The purpose of this paper is to present a method for performing ERP system selection, that allows for the consideration of the effects of interdependence among the decision criteria. Our case study is intended to provide practical experience in applying the method and to provide some indication of its feasibility in practice. We find that the ANP method is practical and it may improve the ERP selection process if it is currently conducted in an ad hoc manner. The results of our case study show that ANP method can produce more relevant information for ERP system selection and decision makers perceive this information as more reliable. It also gets our attention that the additional cost of applying ANP is small, compared to the WSM approach. However, when the number of alternatives and criteria are small, WSM may still be a reasonable method to use, provided that its limitations are take into account and compensated. References Alidi, A. (1996), Use of Analytic Hierarchy Process to Measure the Initial Visibility of Individual Projects, International Journal of Project Management, 14(4), 205 208. Anadalingam, G. and C. E. Olsson (1989), A Multi-Stage Multi-Atribute Decision Model Ofr Project Selection, European Journal of Operational Research, 43, 271 283. Baker, N. and J. Freeland (1975), Recent Advances in R&D Benefit Measurement and Project Selection Methods, Management Science, 21(10), 116 1175. Bernroider, E. and S. Koch (2000), Differences in Characteristics of the ERP System Selection Process Between Samll or Medium and Large Organizations, in Proceeding of the Sixth Americas Conference on Informaton Systems, 1022 1028. Bingi, P., M. K. Sharma, and J. K. Godla (1999), Critical Issues Affecting an ERP Implementation, Information Systems Management, 16(3), 7 14. Buss, M. D. J. (1983), How to Rank Computer Projects, Harvard Business Review, 61(1), 118 125. Enzweiler, A. L. (1997), Software Giants - Get the Big Picture, CMA Magazine, 71(2), 23 28. Journal of e-business, Vol. 5, No. 1, 2003/3 47

A Semi-Structured Process for ERP Systems Evaluation Finnie, G. R., G. E. Wittig, and D. I Petko (1995), Proritizing Software Development Productivity Factors Using the Analytic Hierarchy Process, Journal of Systems and Software, 22, 129 139. Glazer, J. (1999), A Focused Method for Vendor Selection, Manufacturing, 15(2), 33 34. Holland, C. P. and B. Light (1999), A Critical Success Factors Model for ERP Implementation, IEEE Software, 16(3), 30 36. Hong, S. and R. Nigam (1981), Analytic Hierarchy Process Applied to Evaluation of Financial Modeling Software, in Proceedings of the First International Conference on Decison Support Systems, Atlanta. Hwang, C. L. and K. Yoon (1981), Multiple Attribute Decision Making: Methods and Applications: A State of the Art Survey, New York: Springer-Verlag. Illa, X. B., X. Franch, and J. A. Pastor (2000), Formalising ERP Selection Criteria, in Proceedings of the Tenth International Workshop on Software Specification and Design. Kontio, J. (1996), A Case Study in Applying a Systematic Method for COTS Selection, in IEEE Proceedings of ICSE-18, 201 209. Langenwalter, G. A. (2000), Enterprise Resources Planning and Beyond, New York: Lucie Press. Laudon, K. C. and J. P. Laudon (1998), Management Informaton System - New Approaches to Organization & Techology, Prentice hall. Lee, J. W. and S. H. Kim (2000), Using Anaytic Network Process and Goal Progamming for Interdependent Information System Project Selection, Computer & Operations Research, 27(4), 367 382. Min, H. (1992), Selection of Software: The Analytic Hierarchy Process, International Journal of Physical Distribution & Logistics Management, 22(1), 42 52. Sanathanam, R. and M. J. Schniederjans (1993), A Model and Formulation System for Information System Project Selection Method, Computers & Operations Research, 20(7), 755 767. Satty, T. L. (1980), The Analytic Hierarchy Process, New York: McGraw-Hill. Satty, T. L. (1996), The Analytic Network Process, Pittsburgh: RWS Publications. 48

Satty, T. L. (1999), Fundamentals of the Analytic Network Process, Kobe Japan: ISAHP. Satty, T. L. and M. Takizawa (1986), Dependence and Independence: From Linear Hierarchies to Nonlinear Networks, European Journal of Operational Research, 26, 229 237. Journal of e-business, Vol. 5, No. 1, 2003/3 49