DECISION SUPPORT FOR SOFTWARE PACKAGE SELECTION: A MULTICRITERIA METHODOLOGY

5-03-30 INFORMATION MANAGEMENT: STRATEGY, SYSTEMS, AND TECHNOLOGIES DECISION SUPPORT FOR SOFTWARE PACKAGE SELECTION: A MULTICRITERIA METHODOLOGY Farrokh Mamaghani INTRODUCTION Software selection is a critical decision with serious financial and productivity implications. Purchasing software has become an attractive alternative to in-house development, providing economies of scale while altering the risk profile of the implementation project. At the same time, the plethora of powerful and versatile off-the-shelf software packages on the market has made the develop-or-buy decision increasingly complex and challenging. Evaluating and selecting software packages is especially difficult because the process comprises both quantitative and qualitative attributes. Qualitative attributes are attributes that can be identified but cannot be quantified in meaningful (i.e., numerical) terms. Qualitative attributes are important elements in a selection decision, but the lack of a quantified value for them restricts their inclusion in many decision models. Although several decision models and techniques to select microcomputer software packages have been proposed (e.g., linear weighted attribute model, linear assignment model, maximax, elimination by aspects, and lexicographic ordering), they do not incorporate qualitative attributes nor consider multiple criteria in the decision process. This article s presentation of the analytic hierarchy process (AHP) method as a PAYOFF IDEA Qualitative attributes are important elements in decisions regarding microcomputer software package selection, yet the lack of a quantified value for them restricts their inclusion in decision methodologies. The analytic hierarchy process (AHP) method is a decision support tool that lets managers systematically evaluate often conflicting qualitative criteria. It is presented as part of a framework that incorporates business objectives, functional priorities, and an implementation plan into a comprehensive evaluation and selection process. 10/97 Auerbach Publications 1997 CRC Press LLC

multicriteria decision support tool for software package evaluation and selection seeks to remedy that omission. THE SOFTWARE SELECTION PROCESS Software selection essentially involve three basic steps: 1. A framework to incorporate the business objectives in the selection process 2. Determination of functional priorities 3. Preparation of the implementation plan Incorporating Business Objectives Before selecting a specific application software package, an organization should determine which package meets organizational objectives. Typically, information systems are complex, and the effects of implementing them are far reaching. Often there are more opportunities identified than resources available to implement them. IS planning helps minimize the risks of new technology, identify IS opportunities that support the firm s objectives, and rank IS opportunities according to their benefits and the resources available. By definition, IS planning is the process of identifying a portfolio of computer-based applications that will assist an organization in executing its business plans and realizing its business goals (see Exhibit 1). The plan should identify future application areas and the necessary facilities, organization, and personnel to implement the opportunities in each area. It should also highlight innovative opportunities that may produce high returns for little effort or assist in business differentiation. Thus, an IS plan consists of IS opportunities ranked according to the firm s priorities together with proposed actions for realizing the opportunities. Many methodologies are available for performing IS planning, including Holland Systems Corp. s PROplanner Method Guide and IBM Corp. s Business Systems Planning (BSP) approach. Using the IS planning framework, an organization can systematically incorporate its objectives into the software selection process. It is important that a team of people from the functional areas in the organization to be affected by the selected software participate in the determination of business objectives. The team approach not only helps build consensus, but it ensures that all available options are given equal consideration. Determining Functional Priorities The next step is to determine the features and functions the software package should have to meet the requirements established by the users.

EXHIBIT 1 IS Planning Process This is a critical task and mandates that requirements be clearly specified. Specifications lie at the heart of successful software selection. Without them, the target becomes difficult to pinpoint, and it is difficult to assess whether the selection team chose the software that best meets the specifications. The importance of an organizationwide policy regarding the use of application software packages cannot be overemphasized. This means a stated position concerning the desirability of using software packages and the manner in which such packages should be used. Such a policy statement guides an evaluation team as it considers the compromises that users might have to make to employ a software package. The hardware and software policy of an organization has a direct relationship to the choice of application software packages. Usually, the policy is determined by the organizational IS strategy. The evaluation team then restricts its search to vendors offering software packages that will operate in the given technical environment. The organization needs to determine overall vendor and market criteria for software package evaluation and selection. Such criteria might be that each package meets 85% of an application s functional requirements and has been previously installed in at least five organizations.

At this point in the process, functional priorities should be kept at a high level. Otherwise, the criteria become so specific that it is impossible to meet them with any commercially available package. The list of criteria should contain relatively few items and concentrate on the most important ones. These criteria can be selected by the evaluation team in close cooperation with users utilizing a systematic rating or scoring method. The next step is to request and acquire information about different software packages that meet the established functional as well as technical requirements. The availability of several software packages that meet the stated requirements makes the evaluation and selection process a critical one. Because these packages often have different performance ratings, they require a careful and systematic evaluation and selection methodology. THE ANALYTIC HIERARCHY PROCESS The analytic hierarchy process (AHP) makes a significant contribution toward understanding and explaining how decision-makers exercise judgment when confronted with complex, nonprogrammed decisions. 1 By allowing decision-makers to model a complex problem in a hierarchical structure showing the relationships of goals, criteria (i.e., attributes), and alternatives, it allows for the application of experience, insight, and intuition in a logical and thorough way. The AHP methodology is useful for systematically evaluating (often conflicting) qualitative criteria. Similar to other multiattribute decision models, the AHP attempts to resolve conflicts and analyze judgments through a process of determining the relative importance of a set of attributes or criteria. The AHP enables a decision-maker to implicitly develop the trade-off among multiple criteria in the course of structuring and analyzing a series of pairwise judgmental comparison matrixes. The major difference between the AHP and other multiattribute decision models (i.e., utility theory) is that the AHP enables the systematic structuring of any complex multidimensional problem. The attributes of the AHP satisfy the requirements of an effective software selection methodology. The process allows factors to be specified in a multicriteria setting, provides the ability to express the relative importance of the multiple criteria being considered, and uses pairwise comparisons in extracting information. The methodology has been used extensively in practice, including some areas similar to integrated software selection, such as data base management system products, accounting systems, process control, and data acquisition. With the introduction of its interactive PC implementation, Expert Choice (EC), the number and diversity of applications has grown rapidly. IBM used EC on its Application Systems/400 (AS/400) 1. T.L. Saaty, The Analytic Hierarchy Process (New York: McGraw-Hill, 1980).

Project to help win the Malcolm Baldrige Quality Award. General Motors Advanced Engineering Staff used EC to help future car designers evaluate design alternatives, perform risk management, and arrive at the best and most cost-effective automobile designs. Xerox Corporate Research and Technology and the Technology Management groups used EC for R&D decisions on portfolio management, technology implementation, and engineering design selection. EC is also used to help make marketing decisions regarding market matching and customer requirement structuring. STEPS OF THE ANALYTIC HIERARCHY PROCESS Using the AHP to solve a decision problem involves four steps: 1. Setting up a decision hierarchy 2. Collecting input data by pairwise comparisons of decision elements 3. Estimating the relative weights of decision attributes 4. Computing the rating of alternatives Setting up the Decision Hierarchy One of the distinguishing features of the AHP approach is the use of a hierarchic structure to represent the decision problem, independent of problem complexity or the number of criteria. Hierarchical decomposition is one of the most commonly used methods by which decision-makers factor complex problems into more manageable subproblems. As shown in Exhibit 2, the hierarchy generally has at least three levels. At the top of the hierarchy lies the goal of the decision problem. The lower level of the hierarchy contain attributes that contribute to the quality of the decision. The last level of the hierarchy contains decision alternatives for selection choices. EXHIBIT 2 The AHP Decision Hierarchy Collecting Input Data by Pairwise Comparisons of Decision Elements The AHP makes it possible to rank alternative courses of action based on the decision-maker s judgments on intangible qualitative criteria alongside tangible quantitative criteria. The problem hierarchy lends itself to

an analysis based on the impact of a given level on the next higher level. The process begins by determining the relative importance of the criteria in meeting the goals. Next, the focus shifts to measuring the extent to which the alternatives achieve each of the criteria. Managerial judgments are used to drive the AHP methodology. These judgments are expressed in terms of pairwise comparisons (as contrasted with simultaneous comparisons) of attributes on a given level of the hierarchy with respect to their impact on the next higher level. Pairwise comparisons express the relative importance of one attribute versus another in meeting a goal or a criterion. Each of the pairwise comparisons represents an estimate of the ratio of the weights of the two criteria being compared. The use of pairwise comparisons to collect data from the decisionmaker offers some advantages. It allows the decision-maker to focus on the comparison of just two attributes, making the observation as free as possible from extraneous influences. Additionally, pairwise comparisons generate meaningful information about the decision problem, improving information about the decision problem and consistency (compared to simultaneous comparison) in the decision-making process. The pairwise comparison information for each component of the problem is represented by comparison scales. Although many scales could be used for quantifying managerial judgments, the numeric scale in Exhibit 3 is the standard used for the AHP analysis. For example, if a decision-maker believes that attribute A is moderately more important than attribute B, then this judgment is represented by a 3. Judgments are required for all the criterion comparisons and all the alternative comparisons for each criterion. EXHIBIT 3 Pairwise Comparison Scale Intensity of Importance Definition Explanation 1 Equal importance Two attributes contribute equally to the objective 3 Moderate importance of one over Experience and judgment slightly favor one another attribute 5 Essential or strong importance Experience and judgment strongly favor one attribute 7 Demonstrated importance An attribute is strongly favored and its dominance demonstrated in practice 9 Absolute importance The evidence favoring one attribute over another is of the highest order 2, 4, 6, 8Intermediate values between When compromise is needed the two adjacent judgments

Estimating the Relative Weights of Decision Attributes The third step is to determine the relative importance of the alternatives with respect to each criterion (i.e., attribute). The pairwise comparison matrix for a given criterion is used to rank (i.e., establish the relative importance of) the alternatives. This is accomplished using the scaling function previously identified; applying the established attribute weights, the resultant normalized values for the individual alternatives are computed. The process is repeated for every criterion, each resulting in a distinct prioritization of alternatives. Computing the Rating of Alternatives In the fourth step, the results of the two analyses are synthesized to compute the ratings of the alternatives in meeting the goal. After all alternative comparisons are made for each criterion, their relative importance is then elicited from the decision-maker using the same pairwise comparison process used in evaluating the alternatives. When comparing the importance of the individual criteria, the typical question asked of the decision-maker is: In comparing the benefits obtained by attribute A and the benefits obtained by attribute B, which is more important to the entire organization? As before, all possible pairwise comparisons are made, with the responses numerically constructed in another comparison matrix using the same 1 9 scale. Once comparison matrices are constructed for both alternative and criteria comparisons, the final step is to determine the overall ranking of the alternatives. Establishing the overall ranking of the alternatives involves three steps. The first is to determine the relative importance of the criteria using the comparison matrix constructed by the decision-maker. The largest eigenvalue and the corresponding principle eigenvector of this matrix are calculated. The principle eigenvector is normalized such that its entries sum to one. The normalized eigenvector represents the relative importance of the criteria. Finally, the relative importance of the alternatives for each criterion and the relative importance of the criteria themselves are used to determine the overall ranking of the alternatives. Assume that the relative importance of m alternatives have to be established using n criteria. The overall relative importance of alternative j (A j ) is determined as: n j = CiPij i= 1 A where C i P ij = Relative importance of criteria i = Relative importance of alternative j with respect to criteria i

The larger the value of A j, the higher the relative importance of alternative j. Thus, the composite values of A j represent the relative ranking of the alternatives under evaluation. AN EXAMPLE A hypothetical example illustrates how the AHP is used. This example consists of a selection problem involving three competing software packages (S1, S2, and S3). Their prioritization is based on six criteria deemed important for a particular organization: technical requirements, functional requirements, ease of use, vendor support, training time, and price. Exhibit 4 shows the comparison matrix that indicated the results when evaluating the relative importance of the criteria in a pairwise fashion. Exhibit 5 shows the comparison matrices indicating the pairwise evaluation of how the software packages address each criterion. EXHIBIT 4 Criteria Comparison Matrix TR FR EofU VS TT Pr TR 1 1 1 5 9 1 FR 1 1 2 9 9 2 EofU 1 1/2 1 5 9 1 VS 1/5 1/9 1/5 1 2 1/5 TT 1/9 1/9 1/9 1/2 1 1/8 Pr 1 1/2 1 5 8 1 EXHIBIT 5 Software Comparison Matrices S1 S2 S3 S1 S2 S3 TECHNICAL REQUIREMENTS VENDOR SUPPORT S1 1 7 5 S1 1 1/2 1 S2 1/7 1 3 S2 2 1 3 S3 1/5 1/3 1 S3 1 1/3 1 FUNCTIONAL REQUIREMENTS TRAINING TIME S1 1 1/2 3 S1 1 1 1 S2 2 1 6 S2 1 1 1 S3 1/3 1/6 1 S3 1 1 1 EASE OF USE PRICE S1 1 1/3 1 S1 1 1/3 1 S2 3 1 5 S2 3 1 3 S3 1 1/5 1 S3 1 1/3 1 Exhibit 6 provides the relative importance of the software packages by criterion type. For example, using the software comparison matrix for the technical requirements criteria (C1), the normalized eigenvector is calcu-

EXHIBIT 6 Relative Importance (Normalized Eigenvectors) F P EofU VS TT Pr S1 0.072 0.300 0.185 0.240 0.333 0.200 S2 0.649 0.600 0.659 0.550 0.333 0.600 S3 0.279 0.100 0.156 0.210 0.333 0.200 (C1) (C2) (C3) (C4) (C5) (C6) Criteria relative priority: 0.225 0.316 0.199 0.041 0.025 0.194 lated and shown in the technical requirements column. Larger values of the eigenvector indicate greater importance of software packages with respect to the criterion. Thus, S2 best addresses the technical requirements criterion, followed in decreasing order by S3 and S1. This process of calculating the normalized eigenvector is repeated using the software comparison matrices for functional requirements, ease of use, vendor support, training time, and price. Results of these calculations are provided under their respective columns. The results indicate that S2 is the best software alternative when considering the functional requirements criteria, the ease of use criteria, and price. The normalized eigenvector of the criteria comparison matrix is also shown in Exhibit 6. It indicates the relative importance of the criteria based on the decision-maker data. The computational results yield the following: functional requirements is the most important, followed in importance by technical requirements, ease of use, price, vendor support, and training time. Exhibit 7 illustrates the final overall prioritization of the three software alternatives. From this, the order of prioritization would be (from best to worst) S2, S1, and S3. EXHIBIT 7 Composite Prioritization S1:.225 * (.072) +.316 * (.300) +.199 * (.185) + 0.41 * (.240) +.025 * (.333) +.194 * (.200) =.205 S2:.225 * (.649) +.316 * (.600) +.199 * (.659) +.041 * (.550) +.025 * (.333) +.194 * (.600) =.614 S3:.225 * (.279) +.316 * (.100) +.199 * (.156) +.041 * (.210) +.025 * (.333) +.194 * (.200) =.181 Priority Order of Software Packages is S2, S1, and S3. PREPARING AN IMPLEMENTATION PLAN Proper planning to implement a software package is highly critical. Without a detailed plan to guide the implementation activities, the possibility of success becomes slim. Depending on the scope and complexity of the problem, the implementation plan is generally prepared in advance. For a major software package, implementation planning requires the coordination and sched-

uling of several activities and tasks performed by an implementation team comprising managers of affected functional areas, IS staff, vendor representatives, and primary users. As a detailed formulation of how the software package implementation will be achieved, the implementation plan should focus on hardware requirements, necessary modifications, personnel training, documentation preparation, conversion, and postimplementation review. Hardware Requirements In some cases, the new software package may require installation of new hardware or upgrading of existing hardware. In this case, the implementation team should determine the hardware requirements and prepare a timely plan for acquisition and installation. If the existing hardware requires minor upgrades (e.g., adding additional memory), the implementation team should identify such upgrades and plan for them. Software Package Modification The extent of package modification to meet organizational requirements depends on the selected software and ranges on a continuum from no change being permitted to complete modifiability. If the modification is permitted and required, the implementation team should decide how the modifications will be made and by whom. Personnel Training No system can function satisfactorily unless the users and others who interact with it are properly trained. Without an appropriate level of expertise and acceptance by personnel, the system will fail. Personnel training not only increases people s expertise, it also facilitates their acceptance of the new software package an important factor in a package s ultimate success. Training should begin early enough for it to end about the same time the software package becomes operational. Training gives users self-confidence and streamlines and minimizes disruption during early stages of systems operation. Training ranges in scope from a brief tutorial for one user on how to perform a new but simple task to training most, if not all, of the users throughout the organization on a major new software package. Programs that support extensive training agendas include in-house training, vendor-supplied training, and outside training services. Documentation Proper systems documentation is a subject whose necessity most people agree on but then proceed to neglect completely. Documentation benefits managers and supervisors, users, operators, systems professionals,

maintenance programmers, and auditors. It is used for training, instructing, communicating, establishing performance standards, and maintaining the software. Changes in business operations and user requirements necessitate systems and program modifications. Before changes can be made, however, the individual making the change must first understand what the system is supposed to do. Programs written several months or years earlier, or software packages installed recently, must be well documented, because it is easy to forget details in just a few days. Analysis of the total cost of the new system over its life indicates that development costs are less than ongoing operation and maintenance costs. Required maintenance is easier and quicker to perform if documentation is available. Conversion Strategies Conversion is the process of changing from old software to the new software. It requires careful planning. The degree of difficulty and complexity in converting from the old to new system depends on several factors. If the new system is a software package that will run on the present platform, then the conversion is relatively simple. If conversion entails new customized software requiring a new data base, platform, and procedures, it becomes quite involved and challenging. Four different conversion strategies are used: the parallel strategy, direct cutover strategy, pilot strategy, and phased-approach strategy. Parallel Strategy. Both the old system and its potential replacement are run together for a time until everyone is assured that the new system functions correctly. This is the safest conversion approach because in the event of errors, the old system can still be used as a backup. The approach is expensive, however, and additional staff or resources may be required to run the extra system. Direct Cutover Strategy. This strategy replaces the entire old system with the new system on a prescheduled date. This strategy is relatively inexpensive but risky, because if serious problems with the new system occur, the end result will be higher cost than if the parallel strategy was used. Lack of a fall-back system means that dislocation, disruption, and the cost of corrections could be enormous. The Pilot Strategy. The pilot strategy introduces the new system to a limited area of the organization, such as a single department or operating unit. When the pilot version is complete and working smoothly, the system is installed throughout the rest of the organization, either simultaneously or in stages.

The Phased-Approach Strategy. This strategy introduces the new system in stages, either by functions or by organizational units. If, for example, the system is introduced by functions, a new payroll system might first add hourly workers who are paid weekly, followed 3 months later by employees who are paid monthly. If the system is introduced by organizational units, corporate headquarters might be converted first, followed by outlying operating units 3 months later. The Conversion Plan. A formal conversion plan provides a schedule of all activities required to install the new system. The most time-consuming activity is usually the conversion of data files. Data files from the old system must be transferred to the new system, either manually or through special software programs. The converted data file must be carefully verified for accuracy and completeness. Postimplementation Review Postimplementation is the final system implementation task. This process generates its own report, which serves as an addendum to the systems implementation report. Such a review should be conducted at any time from a few weeks to 6 months after systems conversion. A set of interviews and surveys are conducted at any time from a few weeks to 3 months after systems conversion to gather users reactions to the new system in operation. The interviews and observations often suggest enhancements or modifications to be made during systems maintenance. CONCLUSION The framework presented here incorporates business objectives, functional priorities, and an implementation plan into the software evaluation and selection process. It is based on the AHP methodology a decision support tool that lets decision-makers incorporate qualitative as well as quantitative criteria in the decision process. Such criteria include technical requirements, functional requirements, ease of use, vendor support, training time, and price. Use of the model here bolsters previous findings that the AHP methodology is valid, flexible, and easy to apply and does not overlook any significant factor in the selection process. A former senior programmer at Systems Design Concepts, Inc., in Washington, D.C. and systems analyst at the World Bank, Farrokh Mamaghani is an associate professor of MIS at St. John Fisher College in Rochester, NY. His research interests are software evaluation and IT strategic planning.