AHP-based formal system for R&D project evaluation

Transcription

1 Journal of Scientific & Industrial Research Vol. 63, November 2004, pp AHP-based formal system for R&D project evaluation S Suresh Kumar Scientist and Head, Planning and Evaluation Group, Regional Research Laboratory (CSIR), Trivandrum Received 19 March 2004; accepted 23 July 2004 The paper dwells on a judgement model for R&D project evaluation using multifactor criteria based on hierarchic considerations. The idea is to employ formal tools in quantification of subjective evaluations where expert judgement is involved. Comparative evaluations on a priority scale are converted to quantities using the eigen vector concept. This is also the essence of the Analytic Hierarchic Process (AHP) of Satty [Satty T L, The analytic hierarchic process (Mc Graw Hill, New York) 1980] which as a technique is employed for R&D project evaluation. It is argued that AHP is more effective than scoring charts and utility models in priority setting based on judgemental evaluation through peer rating. Keywords: R&D project evaluation, Priority setting, Judgemental models, Analytic hierarchic process, Peer rating Introduction A public funded research laboratory in a developing country like India faces problems with respect to allocation of scarce resources from Government on the large portfolio of projects. Some of these projects are contractual in nature with part of the funds for the same sourced as grants from departmental agencies or Government institutions. In certain cases private or public sector industries may sponsor research by providing entire cost for the project. Even in such cases the project cost projection, particularly in respect of manpower costs are usually on the lower side on account of the exigencies of the situations and expectations in industry circles which make it contingent on public funded research entities to make subsidized projections. Thus, a Government research laboratory in India may have a portfolio consisting of : (i) In-house projects, (ii) Grant-in-aid projects, and (iii) Sponsored research. In-house projects are fully funded by the research institutes through Central Government funds since they are of a basic nature. Industry may not fund such programmes. Nevertheless institutions need to undertake basic research for expertise development and catching up with scientific progress in the field elsewhere. This situation calls for definite criteria for giving priority to the projects so as to facilitate resource allocations. Certain well-defined criteria are discernible in this context. These are applicable to the wide spectrum of projects undertaken and could be categorized as: (i) Organizational, (ii) Technical, (iii) Strategic, and (iv) Financial. With respect to the different types of projects one should put appropriate grading or weightages on these factors through a process that eliminates subjectivity in judgement to the extent possible. Scoring charts are most simple to use but the reliability is in question being subjected to judgenmental errors in grading. Other quantification techniques used are econometric modeling, discounted cash flow/npv/irr calculation, and linear programming methods (non-linear methods). These require more precise data on project benefits and quantifiable socio-economic returns. This is often too premature to hazard at the R&D stage on account of the relative higher levels of uncertainties involved in respect of output measures. Hence, a weighed multiple criteria method was felt to be most suited to the purpose since it helps to overcome the judgemental errors through a systematic methodology 1. Analytic Hierarchy Process (AHP) has a comprehensively structured logical framework to analyze the project in the light of both qualitative and quantitative information. The knowledge of experts in subjective judgement is transformed in to quantitative data 2. Perceived Problems Like in any other research organization the management of the institute (under study) faces crucial decision problems in the selection and ranking of R&D projects.

2 SURESH KUMAR: AHP-BASED FORMAL SYSTEM FOR R&D PROJECT EVALUATION 889 The first problem is that of selecting the most promising project for ensuring the continuation of organizations operations. The number of projects put forward for consideration every year is usually far in excess of the number possible to undertake within the purview of the organizations available resources. Consequently the problem arises in selecting those projects which are likely to be useful to the future viability of the organization. The sources for project proposals are both internal as well as external. The management needed a refined and structured procedure to assist the decision making on the issues mentioned above to account for all the diverse and diffused parameters. The system had to fit within the existing group decision-making process. A decision model was to be developed to assist in project selection and prioritization decisions, along with a conceptual, potential measuring instrument for use by the project management committee. Methods and Models for Evaluation of R&D Projects During the last quarter of twentieth century, management scientists have expended considerable efforts in developing various quantitative and qualitative techniques for R&D management, with special emphasis on project management and particularly on the aspect of priority setting based on multiple criteria. The R&D project selection and prioritization are concerned with the allocation of organizational resources such as, money, skills, and facilities, to a set of proposals for scientific R&D. The decision is important because it may entail a considerable investment commitment. In addition the decision will often have a significant impact on the future financial position and viability of the organization. At the same time, there are typically more projects for R&D than the available resources can support. There are also many inherent uncertainties in attempting to predict project results. For these reasons, among others, it is difficult to structure the alternatives such that an optimal may be made. Many studies were made earlier and formal project selection methods proposed 1,3-5. These methods could be classified into three types according to the basic approach used: Decision Theory approach, the Economic Analysis approach and the Operational Research approach. Each type of approach has certain specific features. The Decision Theory approach suggests certain factors in terms of which a project proposal might be evaluated and derives a scoring model, whereby a rating on an empirical scale is made for each factor considered, and these ratings are combined either by multiplication or as a weighted average or according to heuristic rules to derive a numerical score for each project proposal. The projects then selected are those with the highest scores the number of projects selected being determined by the constraint of the total available budget. The Decision Theory models mostly rely on the subjective or qualitative input variables. The Economic Analysis approach is based on a detailed forecast of the profitability of each project proposal in terms of the investment required and the expected revenue resulting from it. The method for indicating the profitability of a proposal usually makes use of discounted cash flows. The Economic Analysis approach thus assumes that the profit objective is the only objective to be considered in evaluating project proposals and it is dependent on the ability to make a realistic estimate of the anticipated amount of investment required by the project and the revenue expected as a result. For obtaining the information for a benefit cost analysis often presents problems when managers are asked to express their feelings about non-economic factors in money terms. The Operations Research approach is characterized by the use of mathematical programming techniques to optimize the selection of projects by maximizing the total value of the project proposals within the constraints of the total budget and other resources available, the value of each proposal being measured in terms of the financial worth of the projects, which may or may not be discounted. Recently, both binary (0-1) integer and non-liner goal programming formulations have been applied to project selection. Their major improvement over standard 0-1 integer programming techniques is the ability to consider several criteria within the objective function. However, Linear Programming (LP) provides no methods for ensuring that the goals selected adequately reflect the organizational factors related to the decision 6. Applicability of the Methods The quantitative models for project selection and resource allocation appear to the incomplete in the sense that they do not include all the important, relevant aspects of the R&D environments. The problem is characterized by multiple criteria, many of which are not easily quantified, and typical

3 890 J SCI IND RES VOL 63 NOVEMBER 2004 approaches to quantify subjective preferences are far from satisfactory. The R&D process is highly uncertain and unpredictable also. As a result, managers are skeptical of the validity of the numerous data elements, of the various difficult to understand model forms, and of the subsequent allocation recommendations. The general managerial attitude appears to be that qualitative models with their subjective judgments as inputs can assist in the decision making of routine predictable and unpredictable activities 3. Another form of formal project review method involves completion of a checklist or profile chart. Criteria are listed which are believed to be important factors in determining the eventual success or failure of the R&D effort. All projects are then subjectively related on the basis of each criterion listed. The checklist methodology lends itself rapidly to types of information that often prove nearly impossible to conclude in other quantitative model constructions involving non-economic factors such as, social impacts and environmental concerns. Checklists also have certain useful diagnostic properties not found in more general decision models. Particular weaknesses of individual projects are quickly identified by their poor rating on certain criteria. While the simplicity of the approach may be very appealing, it can be ineffective in complex problems. Although many important factors may be included in a checklist, the relevance or importance of each individual factor is left indeterminate. Checklist or profile charts for different projects are difficult to compare because individual criteria are not weighed. Scoring models attempt to remedy this problem by assigning weights to individual criteria and summarizing the results as a single project score. To arrive at a set of criteria weights, it is necessary to extract preference functions from the decision makers. Several methods have been developed for deriving criteria weights. These range from simple rank ordering of criteria to various types of utility functions 7. Scoring models retain the advantages of the checklists and profile charts in terms of their ability to consider a wide range of economic as well as noneconomic criteria. In addition, scoring models make it possible to provide a single number for evaluation of each project. The principal shortcoming of the approach is that the projects score obtained is dimensionless, which makes its use in some situations difficult as opposed to economic or optimization models. Multiple criteria methods on scoring models involve the identification and weighting of several criteria to allow research topics to be ranked on the basis of a composite score. In qualitative multiplecriteria methods, ranking is facilitated by peer group evaluation and rating. Scoring methods are particularly useful when dealing with non- economic criteria and disparate as well as diffuse objective. Weighted criteria methods are also most suited to exante evaluation as in project selection and prioritysetting which is the aim of this study. Here the evaluator is looking at future research operations and the potential benefits they may induce in a semiquantitative manner through peer ratings of an essentially qualitative criteria 8. Analysis of the Problem and Selection of Methodology A study about the characteristics and type of the projects carried out in the institute revealed their complex nature and the uncertainty associated with them. Many facets of the research projects in the laboratory are rather vague and are difficult to formalize and evaluate on an objective basis. It is not easy to define such output measures and progress markers as interim objectives, payoffs, and project costs in advance. Even when the measures are defined the actual outputs are difficult to estimate. Frequently the variance between estimate and achievement is very large, if it were not, the project can hardly be considered as research. This is not simply a matter of forecasting error. Often the estimator cannot even know what methods will be used to reach the research goal. Information about the approach to the goal in most cases is subjective in comparision, e.g. with information about an engineering project. The assumptions in many R&D project management models with the Economic Analysis and Operations Research approach are that the profitability is the main criterion to be considered in evaluating a given project. This does not appear to represent all aspects of the practical situation, although evidently this does represent the overall objective against which the achievement of the output of the total R&D activity might be measured. There are many other factors influencing the decisions regarding the R&D project. The factors are: strategic, environmental, technological, operational, behavioral, and organizational. As revealed by the study of the characteristics of the organization and its objectives, it

4 SURESH KUMAR: AHP-BASED FORMAL SYSTEM FOR R&D PROJECT EVALUATION 891 Intensity of importance on a numerical scale Definition Table 1 The fundamental AHP judgement scale Explanation 1 Equal importance Two factors contribute equally to the objective 3 Moderate importance of one over another Experience and judgement moderately favor one over the other 5 Strong importance One activity strongly over another 7 Very strong importance An activity strongly favored and its dominance demonstrated in practice 9 Extreme importance The evidence favoring one over another is of the highest order of affirmation 2,4,6,8 Intermediate values between two adjacent When compromise is needed judgements Reciprocals If activity i has one of the above numbers assigned to it when compared with activity, j then, j has the reciprocal value when compared with i was concluded that the supporting tools for the project selection and ranking decision-making in the organization must be of a subjective basis. The systematic organization of subjective information on research decisions was proposed for improving the control of the research programs in the institute. Several methods and models for organizing and processing subjective information on multicriteria decision making such as checklists and straight forward point allocation weighted average utility functions 9 were considered and the Analytic Hierarchy Process developed by Saaty 2 is found to be the most promising method as the basis to make the framework for the system 10,11. Analytic Hierarchy Process The Analytic Hierarchy Process (AHP) developed by Saaty 2, allows the decision makers to visually structure a complex problem in the form of a hierarchy having at least two levels; objectives (criteria for evaluation) and activities (products and courses of actions). Each factor or alternative on a given level can be identified and evaluated with respect to other related factor. This ability to structure a complex problem and then focus attention on specific components broadens one s decision making capabilities. Another advantage of AHP is its simplicity, in comparing five conceptually different approaches for determining weights in utility models, Shoemaker and Waid 10 have found that their subjects perceived AHP as the easiest method and the one whose results were most trustworthy. Since 1975, AHP has been applied in various priority setting and resource allocation problems in marketing electric power allocation, conflict resolution, transportation planning, and new product development among others 9. Application of the AHP requires that the problem structure is first disaggregated into a multilevel hierarchy, where each criterion or alternative on a given level is one of the same rough magnitude or importance, and is thought to influence the next higher level. The method centers on determining weights or priorities of a set of criteria in one level of the problem to the next level just above. By repeating this process level by level the matrices summarizing the priorities between levels can be multiplied for determining the priorities of the alternatives at the lowest level according to their influence on the overall goal or focus of the hierarchy. The method of determining the priorities on a given level L=1 to each criteria on the next higher level up is based on a set of pair-wise comparisons. The resulting matrix is a square, positive, reciprocal matrix, i.e., a [ij] = a [ij] - 1 > 0, where a [ij] represents the comparison of the strength of alternative i to alternative j in influencing some stated factor. A fundamental scale consisting of verbal judgements ranging from equal to extreme is used to make the comparisons and it is given in Table 1. This scale has been validated for effectiveness, not only in many applications by many people, but also through theoretical comparisons with a large number of other scales 2. The largest eigen value of the pair-wise comparison matrix is computed, and its associated eigenvector represents priority (ranking).

5 892 J SCI IND RES VOL 63 NOVEMBER 2004 Considerable attention in an AHP application is also placed on the measurement of inconsistency in human judgement. Inconsistency results, if a [ij] a [jk] is not equal to a [jk] for some i,j,k. Various methods are available for measuring the inconsistency of pair-wise comparison matrix; small deviations in the a [ij] away from the implied ratio scale lead only to small deviations in the priority vector, and to a generally stable solution. This result is due in part to the fact that the pair-wise comparison matrix is positive reciprocal, and that redundant information relating to priorities is present in each pair-wise comparison matrix. This approach differs from that of standard scoring models, since the weights provided for the ratings of each sub-criterion are not based on arbitrary scale, but utilize a ratio scale for human judgments. Also, a hierarchy is not the traditional decision tree. Each level may represent a different cut at the problem. One level may represent social factors and another technical factors to be evaluated. Further, a decision maker can insert or eliminate levels and elements as necessary to clarify the task of setting priorities or to sharpen the focus on one or more parts of the system. Elements that have a global character can be represented at the higher levels of the hierarchy, others that specifically characterize the problem at hand can be developed in greater depth. The task of setting priorities requires that the criteria namely, the properties for features of the alternatives being compared, and the alternatives themselves are gradually layered in the hierarchy so that it is meaningful to compare them among themselves in relation to the elements of the next higher level 2. Finally, after judgements have been made on the impact of all the elements and priorities have been computed for the hierarchy, as a whole, sometimes, and with care the less important elements can be dropped from further consideration because of their relatively small impact on the overall objective. The priorities can then be recomputed throughout, either with or without changing the remaining judgements. The framework for project selection system was developed after considering the organizational context. As explained earlier, it involves the availability and reliability of data for measuring costs and benefits the statement of organizational and project goals, and the structure of R&D and supporting groups, among others. It is evident that project selection is based on economic as well as social benefit-cost analysis. This involves the consideration of both quantitative and qualitative factors. Based upon an extensive literature survey, a list of factors was identified which were deemed critical in influencing the decision to select a portfolio of R&D projects 12. Then, after detailed discussions with the management personnel at the Institute the factors were short listed and modified so that only those variables that the management felt more important, and to which they can provide hard data or firm opinions, were included. A method using Analytical Hierarchy Process 2 (AHP), for the measurement and aggregation of the various project selection criteria so as to prioritize and rank many projects, was developed 13. The key factors finally short listed for the project selection model fell into four principal categories, viz, Organizational, Technical, Strategic and Financial (Table 2) Model Formulation The AHP model developed for the project selection system is presented in Fig. 1. In the first or top level is the overall goal of the system to obtain the best portfolio of projects. The second level is made up of the four factors, which contribute to the goal, and the third level is made up of the criteria constituting each Table 2 Factors relevant to project selection decision (I) Organizational factors 1 Availability of human expertise to carry out the project in the organization 2 Adequacy of equipment and facilities 3 In-house availability of technology 4 Availability of material resources and consumables (II) Technical factors 1 Probability of technical success 2 Attractiveness of technological route 3 Anticipated completion time 4 Extent of innovation in the project objective 5 Extent of tie-in with existing projects 6 Technological relevance of the project (III) Strategic factors 1 Relevance of project objectives in the social context 2 Clarity of definition of project objectives and its ultimate benefits (project mission) 3 Utility of regional resources 4 Reputation of project leader 5 Anticipated change of commercial success (IV) Financial factors 1 Financial feasibility of the project 2 Commercial sponsorship for the project 3 Aids or collaboration for the project from outside agencies

6 SURESH KUMAR: AHP-BASED FORMAL SYSTEM FOR R&D PROJECT EVALUATION 893 factor for project proposal evaluation. The pair-wise comparison of the four factors in relation to the focus is done first to reflect the importance of each of the factors. Next, pair wise comparison matrices are developed to reflect the importance of each criterion in relation to the four factors on an individual basis. Next level is the evaluation of candidate projects. If the number of projects proposed for consideration is small then the projects could be pairwise compared with respect to each criterion. However, when the number of projects is large, that method is generally infeasible. For example, if there are 20 R&D project proposals, n (n-1) / 2=190 pairwise comparisons are required for each of the 18 criteria listed. The explosion in the number of required comparisons is the basic criticism of the AHP method. The absolute measurement method of Fig. 1 AHP model for project selection system the AHP overcomes this problem as the alternatives are not pair-wise compared in this method. Instead, it is enough to rate projects against each criterion on the grading scale. A grading scale with five elements was devised for rating the candidate projects, as shown in Fig. 2. A hypothetical example of project evaluation by a single evaluator using the AHP-based project selection model is presented here. The procedure starts with the elicitation of pairwise comparison judgements to set priorities for the criteria used in evaluation. The first level elements are arranged into a matrix and the pair-wise comparisons are done using the fundamental AHP judgement scale given. The question to be asked for each entry in the matrix of comparison is when the element on the left hand side is compared with the element on the top, how much more (or less) important it is with respect

7 894 J SCI IND RES VOL 63 NOVEMBER 2004 Fig. 2 AHP grading scale for the project selection model to the overall goal of selecting the best portfolio of projects? If the element on the left hand side is more important, it is given an integer value 1-9 corresponding to its importance. If it is less important, then it is given the corresponding reciprocal value. The matrix of pair-wise comparisons for level 2, along with the resulting priorities, is given in Table 3. For example (referring to the scale), organizational factor (factor 2.3) is attaining the stated goal and hence assigned a value of 3. In turn, the comparison score of factor 2.3 to factor 2.1 has the reciprocal value of 1/3. The comparison of an item to itself is given the value 1. Of all the pairs in the matrix, only the pairs written above the diagonal need to be compared, as the elements below the diagonal of the matrix are obtained by taking reciprocal of the upper ones. As explained the AHP determines the priorities of each factor or the importance or weight that should be given to each factor, by analyzing such judgemental matrices, using the mathematical theory of eigen values and eigen vectors 10. AHP interprets the eigen vector associated with the largest eigen value as the priorities that indicate the importance of each alternative in accomplishing the objective. The eigen vector [P] for the factors are computed by the standard matrix method for computing eigen vector and associated maximum eigen value (or the geometric method which nearly approximates to the same). The priority or Table 3 Hypothetical AHP comparison matrix for level 2 Level Eigen vector Priority vector [Pi] [Pi]=[Pi]/ [P] ½ ½ 1 3 ½ /3 1/ / P= = weight [Pi] given to the i the factor is obtained by normalizing the Pi (i.e, transforming them so that their resultant sum equals unity); i.e, [Pi]=[Pi] / [P], where [P] is the sum of [Pi]. As an alternative, for a square matrix with n rows and columns, the required priority eigen vector can approximately be calculated as follows. For each row I of the matrix, the product of the elements in that row is taken and the corresponding geometric mean is found out. Normalization of the resulting values will yield the approximate priority eigen vector. Similarly the analysis is done for the third level in the hierarchy and the comparison matrices are obtained (Tables 4-7) Composite hierarchical priorities for the criteria are obtained by multiplying each of the level 3 priority by the corresponding level 2 priority as shown in Table 8. For example For criterion 3.1.1, p = Next step is to establish the composite priorities for the candidate projects if the number of projects is small (seven or less). Projects can be pair wise compared with respect to each criterion (level 3) and the relative composite priorities can be found out. However, when the number of projects is large, it is generally infeasible. Hence the absolute measurement is applied, using the grading scale.

8 SURESH KUMAR: AHP-BASED FORMAL SYSTEM FOR R&D PROJECT EVALUATION 895 Table 4 AHP comparison matrix for level 3.1 Level Priority /3 1/ /3 ¼ 1/ Table 5 AHP comparison matrix for level 3.2 Level Priority /3 1 ½ ¼ ½ ½ ½ 2 1 ½ 1/3 1/ / ¼ 2 3 1/ ½ 2 3 ½ Table 6 AHP comparison matrix for level 3.3 Level Priority ½ /3 1/3 1 1/5 1/ /3 1/ /3 ½ 5 1/ Table 7 AHP comparison matrix for level 3.4 Level Priority ¼ ½ / Table 8 AHP comparison matrix for level 5 Level 3.1 Very Poor Poor Satisfactory Good Excellent Priority Very Poor 1 1/3 1/5 1/7 1/ Poor 3 1 1/3 1/5 1/ Satisfactory /3 1/ Good Excellent Priorities are assigned to the grades through paired comparisons (Table 8). It is assumed that the grades are the same for each of the parent criteria. Candidate projects are evaluated for each criterion by identifying the grade, which best describes it (Table 9) and the score obtained for each project is calculated. Thus the priority scores for the project proposals are obtained, and they are ranked based on their magnitude. Conclusions Though OR and econometric methods are used in research evaluation, weighted multi-criteria models have been found to be more suited to the predominantly high-risk, low certainty profiles of research situation in public funded laboratories. Subjective judgements on expected outcomes and intended impacts are typical of research evaluation exercises, aimed at priority-setting and project portfolio selection by management. AHP through its structured hierarchy of decision levels and pair-wise comparison of elements for value judgement is more effective than utility models and scoring charts in working at semi-quantitative data as realistic inputs to the priority-setting agenda. They help to overcome in

9 896 J SCI IND RES VOL 63 NOVEMBER 2004 Table 9 AHP evaluation matrix for hypothetical projects CRITERIA PROJECT PROPOSALS No wt G P G P V P P P S P V S S V G S S P S S G S S P S G S P V S S S S S S S G P S G S G P V G S G P S P S S G P S S S P G P S P G P S P G S G S S S P P S P G P G S G S S P S P G S G S S V G P G V S V V G P V Note; V- very good, P- Poor, S- Satisfactory, G-Good, and E- Excellent For example, for project 1, Total score = (0.108) (0.262) + (0.114)( 0.063) + ( 0.049)(0.129) + (0.026) (0.129) + (0.075)(0.129) + (0.014)(0.129) + (0.020) (0.262) + (0.050)(0.129) + (0.030)(0.063) ) (0.129) + (0.069)(0.129) + (0.050)(0.129) + (0.011) (0.262) + (0.037)(0.063) + (0.025)(0.262) + (0.040) (0.129) +(0.181) (0.129) + (0.069)(0.129) a significant way the fuzzy nature of quantitative information related to deliverable, logistics, and outcome. In the resource constrained situation of the developing countries, AHP provides a vital tool to select and rank projects based on judgemental evaluation through peer ratings. It is also found to be most amenable to computerized decision support system implementation. Acknowledgements The author acknowledges the help rendered by Mr N P Manojkumar, M Tech scholar in literature search and data analysis. The help rendered by Mr M P Varkey in preparing the chart, graphics and the electronic manuscript is also acknowledged. References 1 William D J, A study of decision models for R&D project selection, OR Quarterly, 20 (3)(1969) Satty T L, The analytic hierarchic process (Mc Graw Hill, New York) Baker N R, R&D Project Selection Models, An assessment, IEEE Trans. Engg. Mgmt, 21 (4) (1974) Costello D, A practical approach to R&D Project selection, Technol Forecast Soc.Change, 23(1983) Mills B & Kamau M, Methods of prioritizing research options (ISNAR, The Hagiee)1998, pp Libertore M J & Titus G J, The Practise of Management science in R&D project, Manage Sci 29 (8) (1983) Monteith G Dm, R&D Administration (Hiffee Books Ltd, London)(1969). 8 Hartwich F & Janssen W, Setting research priorities, Res Evaluat, 9(3) (2000) Gass S I, Decision making models and algorithms (John Wiley & Sons Inc, New York) Schoemaker P J H & Waid C C, An experimental comparison of different approaches to derermining weights in additive utility models, Manage Sci, 28(2) (1982) Libertore M J, An extension of the Analytic Hierarchy Process for industrial R&D project selection and research allocation, IEEE Trans Engg Mgmt, 34(1)(1987) Suresh Kumar S, Ganesh L S & Manoj Kumar N P, AHP based DSS, M Tech project thesis, IE & M Division, IIT, Madras, (1992). 13 Pinto J K & Selvin D P, Critical factors in successful project implementation, IEEE Trans Engg Mgmt, 34(1)(1987)