An AHP/DEA method for measurement of the efficiency of R&D management activities in universities

Intl. Trans. in Op. Res. 11 (2004) 181 191 INTERNATIONAL TRANSACTIONS IN OPERATIONAL RESEARCH An AHP/DEA method for measurement of the efficiency of R&D management activities in universities Y.J. Feng a,h.lu a and K. Bi b a Management School of Harbin Institute of Technology, Harbin 150001 and b Science & Technology Management Institute of Harbin University of Science and Technology, Harbin 150080, China E-mail: fengyi@hope.hit.edu.cn [Feng]; lvhui@263.net [Lu] Received 25 September 2002; received in revised form 6 April 2003; accepted 30 June 2003 Abstract In order to develop a better tool for the assessment of the management performance of research and development (R&D) activities in research-oriented universities, a combination of analysis hierarchical process (AHP) and data envelopment analysis (DEA) is proposed for the assessment of the efficiency of R&D management activities in universities. The measure consists of the measurement of a university s previous and present R&D strength by AHP and the assessment of the relative efficiency of its growth in R&D strength against those of other universities by DEA, in which the management basis of the measured universities is taken into consideration. The application of the measure to assess the R&D management efficiency of 29 universities in China indicates the universities which have improved their management work achieved a high efficiency value regardless of whether their original R&D strengths were strong or weak. Such a measure is proved to be helpful for motivating the universities to keep on improving their R&D management. Keywords: Data envelopment analysis; management efficiency; R&D activities; analysis hierarchical process; ranking universities Introduction After the education reform, many colleges in China have expanded into comprehensive universities with departments across a wide variety of fields to further enhance their research and development (R&D) strengths. So the universities have to face up to the challenge of enhancing their management performance in R&D activities. As the assessment of the inner management work of R&D activities in universities helps to motivate the managerial staff of the universities to improve their work and then to enhance the R&D strengths of universities at large, r 2004 International Federation of Operational Research Societies. Published by Blackwell Publishing Ltd.

182 Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) 181 191 a practical and reasonable measure is developed for the assessment of the efficiency of the management work at R&D activities in universities in this paper. The analysis hierarchical process (AHP), a comprehensive strength-evaluation method fails to measure the performance of management work in many cases because it neglects the influence of the diversity in management basis of the measured units on the evaluation result, which makes the results less efficient and objective. The same problem exists with the data envelopment analysis (DEA) method, a relative efficiency evaluation method of a group of units. In this paper, we propose a two-stage measure which combines the AHP and DEA to assess the management efficiency of R&D activities in universities. In the first stage, a university s previous and present R&D strength is measured by AHP and the relative efficiency of its growth in R&D strength against those of other universities is assessed by DEA. Thus the efficiency of R&D management activities in a university is measured by its growth rate in R&D strength according to other measured universities. In this way, the measure overcomes the shortcoming of AHP and DEA itself when used for assessing the performance of management work. Such a measure is helpful for motivating the measured universities to improve their inner management work at R&D activities and to explore efficiency ways to enhance their R&D strengths. This paper is organized as follows. In the second section we will review the evaluation work of universities. The mathematical model for our proposed measure is presented in the third section. In the fourth section, we apply the measure to assess the efficiency of R&D management activities in 29 universities in China to give a further illustration of the method. Literature review Since the US news and World Report published its ranking result for US universities for the first time in 1983, ranking activities for universities have become popular worldwide and have, to a great extent, promoted the development of universities (Webster, 2001). With the transference of a university from the center of education to centers of education and R&D (Mansfield, 1998), rating activities of universities have extended from the assessment for teaching quality to comprehensive strength, in which R&D standing takes an increasing heavy weight (Ahn, Charnes and Cooper, 1998; Johnes, 1988). Factors contributing to the success of R&D activities in universities include policy tendency (Lowry, 2001; Carac, Conceic and Heitor, 2000), good organization arrangement (Lindsay, 1982), pursuit of a good reputation for the university authority (Baden-Fuller, Ravazzolo and Schweizer, 2000), inner innovation motivation in universities (Miyata, 2000). However, the key factor which will bring out an increase in strength is the efficient inner management of R&D activities. Many methods are used for the assessment of the R&D activities in universities, among them the analysis hierarchical process (AHP), factor analysis and cluster analysis are also very popular (Xu and Wang, 1998). These provide a description of the strength standing of the measured universities. However, for the purpose of analyzing the management work at R&D activities in universities, efficiency analysis is usually a practical way. The DEA method provides a feasible and simple way for relative efficiency evaluation of public sectors characterized by multi-input and multi-output (Charnes, Cooper and Rhodes, 1978) and is widely used for the assessment of

Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) 181 191 183 universities, from research efficiency of universities (Ahn, Arnold, Charnes and Cooper, 1989; Avkiran, 2001; Coelli, 1996) and teaching efficiency of departments ( Johnes and Johnes, 1993; Tomkins and Green, 1988) to scaling graduate school (Stern, Mehrez and Barboy, 1994). However, few of them focus on the assessment of performance of management work in universities, which will provide an insight into the potential growth in R&D strength of universities, and will be helpful for enhancement of R&D strength from a strategic development view. For this purpose, we present our measure for the efficiency of management work. Model The first stage: AHP method AHP, developed by Satty in 1980 (Saaty, 1980), is a simple and feasible multi-objective evaluation method widely used for multi-object evaluation activities. It is designed for subjective evaluation of a set of alternatives based on multiple criteria organized in a hierarchical structure. At the top level, the criteria are evaluated and at the lower levels, the alternatives are evaluated by each criterion. The decision-makers assess his evaluation separately for each level and sub-level subjectively. By creating a pairwise comparison matrix, his subjective evaluation for every pair of items is assessed (Dyer, 1990). In the first stage of our measure, by AHP, we get the weight of each indicator of a hierarchical indicator system established for the assessment of the R&D strength of a group of researchoriented universities. The weighted sum of all indicator data of a university is treated as its strength in the data time. Here, we use the common title decision-making unit (DMU) for a measured university. Then we give the following definitions: Definition 1: the historical or previous strength of a DMU is its Reference Index (RI) Definition 2: the present or later strength of a DMU is Present Index (PI) Definition 3: the RI and PI constitutes the Index State of a DMU, represent by (RI, PI) The second stage: DEA method Suppose there are n DMUs. DMU j represents the jth DMU. x j represents the RI of DMU j and y j represents the PI of DMU j. Then (x j,y j ) represents the IS of DMU j. We define T as the Index Possibility Set made up by the DMUs. The definition of T is shown in (1). ( ) T ¼ ðx; yþj Xn l j x j 4x; Xn l j y j 4y; Xn l j ¼ 1; l j X0; j ¼ 0; 1; 2; :::; n ð1þ j¼0 j¼0 j¼0 wherein ðx 0 ; y 0 Þ¼ð0; 0Þ. Obviously, T is a convex set, viz. if ðx 0 ; y 0 Þ2T, ðx 00 ; y 00 Þ2T, then ðlx 0 þð1 lþx 00 ; ly 0 þð1 lþy 00 Þ2T; 04l41.

184 Using the following single input to single output DEA model presented by Banker, Charnes and Cooper (1984) as shown in (2) on T, we can get the relative efficiency of each DMU. max Z s:t: Xn l j a j 4x j0 X n j¼0 X n j¼0 j¼0 l j y j Xzy j0 l j ¼ 1 l j X0; j ¼ 0; 1; 2; :::; n Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) 181 191 wherein x j is the RI of DMU j and y j is the PI of DMU j, j ¼ 0; 1; 2:::; n, x 0 ¼ 0; y 0 ¼ 0. Suppose Z 0 is the optimal value of DMU j by (2). Set Z j the management efficiency (ME) of DMU j, then Z j ¼ 1=Z 0 100% ð3þ If Z 0 5 1, the corresponding DMU is on the frontier of T. Suppose z 0 is the optimal value of ðx j0 ; y j0 Þ by (2), set x j0 ¼ x j0 ; y j0 ¼ z 0 y j0, it can be seen that ðx j0 ; y j0 Þ is right on the frontier of T, as shown in Fig. 1. We refer ðx j0 ; y j0 Þ as the projection point of ðx j0 ; y j0 Þ on the frontier of T. Then the ME of ðx j0 ; y j0 Þ is Z j0 ¼ y j0 =y j0 100% ¼ 1=Z 0 100% ð2þ Illustration of the model The principle of the above measure can be illustrated by Fig. 2. In Fig. 2, A, B, and C are three DMUs respectively. Their corresponding RIs are along the X axis as input and their PIs are along the Y axis taken as output. Suppose A and C are two DMUs whose growth in strength is optimal at its standing. As the RI of B is higher than that of A but lower than that of C, as shown in Fig. 2, B lies between A and C. B is under the linked line from A ( x j0, y j0 ) C D Y : Present Index A B ( x, y j0 T j0 ) 0 X : Reference Index Fig. 1. Illustration of the model.

Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) 181 191 185 Y: Present Index A ( x A, y A) B ( x B, yb ) B( x B, yb C x C, y ) ( C ) 0 xa xb X : Reference Index x C to C. This means the efficiency of the management work in B is less efficient as it did not enhance its strength enough to reach the optimal level. In other words, if B is efficient as A or C, it should reach to the strength level line from A to C with its RI. Therefore B is inefficient in the measure and its efficiency value is measured by the ratio of its PI to the corresponding point in line AC, B 0 in this example. Measurement of the management work of R&D Activities at 29 universities in China Evaluation indicator system and data A three-tier indicator system is established for the evaluation of comprehensive R&D strength of universities in China, in which the R&D input and R&D output are considered together. The R&D input of a university is measured by its personnel structure and R&D expense. And the R&D output is usually measured by the number of published papers, number of research and applied projects. Moreover, patents and identified R&D projects are also important for the assessment of the R&D standing of a university. Using a questionnaire from some experts from the Education Appraisal Institute affiliated to the National Education Ministry of China, we get the relative weight for each indicator in the evaluation index system through AHP, as shown in Table 1. The universities being measured in our example are 29 universities which are affiliated to the National Education Ministry of China and the indicator data has been selected from the Chinese Higher Education Statistical Data. The weighted sum of all indicator data of a university represents its R&D strength standing in the data time. In this case, we use the weighted sum of all indicator data of a university in 1996 as its RI (namely, its management basis) and that of it in 1997 as its PI (its performance of management work in 1997). In this way, we can get the RIs and PIs for all measured universities, shown in Table 2. Then using the DEA model above, with the RIs of the 29 universities as input and the PIs of them as output, we can get their MEs in 1997, as shown in Table 3. Result and analysis Fig. 2. Illustration of the principle of the measure. Using RIs of all measured universities as input X and PIs of all measured universities as output Y, we can produce the graph in Fig. 3, which provides an insight into our measure.

186 Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) 181 191 Table 1 Indicator system for evaluating the R&D management efficiency of universities in China First-tier indicator Second-tier indicator Third-tier indicator Weight Full-time R&D personnel Number of academic faculty Professors 1.84 Professors 1.22 Lecturers 0.55 Teaching assistants 0.36 Others 0.17 Number of other faculty Senior faculty 1.34 Junior faculty 0.54 Primary faculty 0.19 R&D expense 6.21 Project R&D project Basis research projects 2.95 Fruits of science and technology Application research 1.10 projects Experimental and 0.61 developmental projects Technology service project Application of other R&D projects 1.18 0.37 Publication International 7.45 publications National publications 1.76 Regional publications 0.69 Monograph 0.77 Identified product International level 2.75 National initiation 1.16 National leading 0.63 Others 0.27 Patent Inventions 3.7 New practical 0.79 application Appearance design 0.32 5.79 International communications People attending international conference People attending 3.51 conference held at home Visiting scholar 1.76 Postgraduate abroad 0.87 Awards National awards 24.9 Provincial or departmental 9.05 awards Regional awards 3.28 Income of technology transference 11.93 The origin (O) and the four optimal universities, North-east Normal University (5.8, 12.68), Fudan University (28.53, 41.48), Peking University (50.51, 51.02), and Zhejiang University (64.9, 56.7) construct the optimal management frontier ONFPZ of our case. In the measure the ME of

Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) 181 191 187 Table 2 Reference indexes and present indexes of 29 universities in China. University RI (Comprehensive strength standing of 1996) PI (Comprehensive strength standing of 1997) RI Order PI Order Peking University 50.51 2 51.02 2 Beijing Normal University 9.95 25 11.98 24 Nankai University 22.83 13 21.33 15 Tianjing University 37.59 5 37.4 6 Dalian Science and Engineering University 23.41 12 20.1 17 Jilin University 16.39 19 20.78 16 North-Eastern Normal University 5.8 28 12.68 23 Fudan University 28.53 10 41.48 4 Tongji University 28.38 11 36.41 8 Shanghai JiaoTong University 30.31 9 36.52 7 East China University of Science and Technology 16.73 17 24.84 12 East China Normal University 6.49 27 7.47 26 Nanjing University 39.51 4 29.83 11 South-East University 33.65 7 34.1 10 Zhejiang University 64.9 1 56.7 1 Xiamen University 15.08 20 14.68 21 Shangdong University 22.08 14 19.13 18 Qingdao Sea University 11.16 23 7.89 25 Wuhan University 13.95 22 16.12 20 Huazhong University of Science and Technology 45.1 3 39.13 5 Central China Normal University 7.8 26 4.8 27 Zhongshan University 18.22 16 16.49 19 South China University of Science and Technology 16.7 18 22.87 14 Sichuan Union University 31.18 8 34.5 9 Chongqing University 20.69 15 23.76 13 South-West Normal University 14.43 21 2.63 29 Xi an Jiaotong University 35.59 6 41.61 3 Shanxi Normal University 2.95 29 2.75 28 Lanzhou University 10.65 24 13.71 22 a university in the group is measured by the ratio of its PI to its corresponding projection point on the frontier ONFPZ. Take Fudan University, Tongji University, and Xi an Jiaotong University ( points F, T, and X in Fig. 3) for example. It is obvious that the ME of Fudan University is higher than that of Tongji because with very close R&D strengths in 1996 (RIs in our measure), Fudan achieved 41.48 in 1997, much higher than Tongji s 36.41. The situation is quite different when comparing Fudan to Xi an Jiaotong University. Although the strength of Xi an Jiaotong University in 1997 is a little higher than that of Fudan, 41.6 to 41.48, the ME of Xi an Jiaotong is only 93.41, lower than Fudan s 100 because with quite better original strength standing, 35.59 in 1996, Xi an Jiaoting s growth in strength is much smaller. This means the management work in Xi an Jiaotong

188 Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) 181 191 Table 3 R&D management efficiency of 29 universities in China in 1997. The measured universities Strength in 1997( PI) Management efficiency PI Order ME Order Peking University 51.02 2 100 1 Beijing Normal University 11.98 24 66.78 18 Nankai University 21.33 15 62.26 20 Tianjing University 37.40 6 82.36 10 Dalian Science and Engineering University 20.10 17 57.44 23 Jilin University 20.78 16 79.62 13 North-Eastern Normal University 12.68 23 100 1 Fudan University 41.48 4 100 1 Tongji University 36.41 8 88.18 7 Shanghai JiaoTong University 36.52 7 86.43 8 East China University of Science and Technology 24.84 12 93.63 5 East China Normal University 7.47 26 55.11 25 Nanjing University 29.83 11 64.5 19 South-East University 34.10 10 78.03 14 Zhejiang University 56.70 1 100 1 Xiamen University 14.68 21 60.07 21 Shangdong University 19.13 18 57.43 23 Qingdao Sea University 7.89 25 40.52 27 Wuhan University 16.12 20 70.07 17 Huazhong University of Science and Technology 39.13 5 80.40 12 Central China Normal University 4.80 27 31.55 28 Zhongshan University 16.49 19 58.03 22 South China University of Science and Technology 22.87 14 86.33 9 Sichuan Union University 34.50 9 80.93 11 Chongqing University 23.76 13 75.32 15 South-West Normal University 2.63 29 11.14 29 Xi an Jiaotong University 41.61 3 93.41 6 Shanxi Normal University 2.75 28 42.64 26 Lanzhou University 13.71 22 72.83 16 Fig. 3. Illustration of management efficiency of 29 universities in China.

Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) 181 191 189 Table 4 Data of indicator of Beijing Normal University, Northeast Normal University and East China Normal University. Indicator Beijing Normal University (9.95, 11.98) North-east Normal University (5.8, 12.68) Lanzhou University (10.65, 13.71) Weight 1996 1997 Growth Rate (%) 1996 1997 Growth Rate (%) 1996 1997 Growth Rate (%) National awards 24.90 0 0 F 0 2 83 94 13.25 Technology transfer 11.93 32 363 1034.38 120 500 316.67 221 259 17.19 Departmental awards 9.05 6 12 100.00 1 11 1000.00 97 121 24.74 International publications 7.45 179 160 10.61 63 77 22.22 99 115 16.16 R&D expense 6.21 15060 15263 1.35 8843 8540 3.43 6 7 16.67 People attending international conference 5.79 48 68 41.67 21 30 42.86 73 92 26.03 Invention patent 3.70 1 4 300.00 1 0 100.00 146 158 8.22 People attending conference held at home 3.51 63 48 23.81 34 2 94.12 63 77 22.22 Regional awards 3.28 0 1 F 2 0 100.00 115106 126267 9.70 Basic research 2.95 208 185 11.06 70 71 1.43 125 146 16.80 Identified project of international level 2.75 2 4 100.00 0 2 410 479 16.83 Professor 1.84 87 95 9.20 65 65 0.00 304 355 16.78 National publications 1.76 403 443 9.93 66 276 318.18 357 418 17.09 Visitor scholar 1.76 40 47 17.50 29 17 41.38 83 175 110.84 Senior faculty 1.34 54 52 3.70 30 30 0.00 222 244 9.91 Associate professor 1.22 141 130 7.80 104 104 0.00 1032 1217 17.93 R&D fruit 1.18 5 11 120.00 5 5 0.00 185 245 32.43 Fruit of national initiation 1.16 1 2 100.00 6 1 83.33 31 63 103.23 Applied research 1.10 44 50 13.64 143 147 2.80 23 24 4.35

190 University is less efficient. Or in other words, under the same management, the R&D strength of Xi an Jiaotong University in 1997 should have achieved 44.41 rather than 41.48. Using Northeast Normal University (N in Fig. 3) as another example, the comprehensive strength of N in 1997 is far lower than Fudan University, 12.68 to F s 41.48. But its R&D strength has a remarkable increase, from 5.8 in 1996 to 12.68 in 1997. Such a growth rate is much higher than those of other universities with similar R&D strengths in 1996, for example Lanzhou University, L (10.65, 13.71) and Beijing Normal University, B (9.95, 11.98). Therefore, the ME of Northeast Normal University is as high as 100. Table 4 lists data of some key indicators (which are with weight more than 1) of the universities. From Table 4, we can see that some important indicators of Northeast Normal University generally achieved a higher increase from 1996 to 1997. Some indicators even get a breakthrough from zero. Although the important indicator data of Beijing Normal University and Lanzhou University has also increased to some extent, their growths in strength are far lower than Northeast Normal University. Thus their MEs are 66 and 78 respectively. In the measure, the efficiency of a university is assessed by its corresponding criteria defined by the behavior of a group of universities. Thus the measure fulfills our goal to assess the management work with full consideration of the original standing of the measured universities. By discussion of the result with some experts from the Education Appraisal Institute, their reflection on the result is positive. Most of them hold the idea that the result is reasonable and can provide much information about the inner management work at R&D activities in the measured universities. They agree that the measure can dig out more useful information which is helpful for the improvement of their management work and motivate the university to keep on improving their work. Conclusion As the assessment of the inner management work at R&D activities in universities is helpful for motivating the managerial staff of the universities to improve their work from a long strategic development view, a two-stage method which combines the existing methods of AHP and DEA method is developed in this paper. In the measure, the original R&D strength of a university is taken as its management basis for the assessment of its management work and its growth rate in R&D strength against those of other universities is taken as its efficiency of management work. The application of the measure to assess the efficiency of R&D management work of 29 researchoriented universities in China shows the efficiency values of universities varies a lot from their present R&D strengths. The universities with high efficiency value in the case improved their work a lot, which resulted in a higher increase in their R&D strength than those of other universities. The measure is proved to be reasonable and practical for the assessment of management work and it can also provide insight into the evolution of the R&D management work in universities when it is used for a long period of time. Acknowledgements Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) 181 191 This paper is taken from the project Study on Appraisal Theory and Methods of Performance Management (No. 70131010). Supported by the National Science Foundation of China.

References Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) 181 191 191 Ahn, T., Charnes, A., Cooper, W.W., 1998. Some statistical and DEA evaluations of relative efficiencies of public and private institutions of higher learning. Socio-Economic Planning Sciences, 22(6), 259 269. Ahn, T., Arnold, V., Charnes, A., Cooper, W.W., 1989. DEA and ratio efficiency analyses for public institutions of higher learning in Texas. Research in Governmental and Nonprofit Accounting, 5, 165 185. Avkiran, N.K., 2001. Investigating technical and scale efficiencies of Australian universities through data envelopment analysis. Socio-Economic Planning Sciences, 35, 57 80. Baden-Fuller, C., Ravazzolo, F., Schweizer, T., 2000. Making and measuring reputations: the research ranking of european business schools. Long Range Planning, 33, 621 650. Ball, D.F., 1997. Quality measurement as a basis for resource allocation: research assessment exercises in United Kingdom universities. R & D Management, 27(3), 281 293. Banker, R.D., Charnes, A., Cooper, W.W., 1984. Some models for estimation technical and scale efficiencies in data envelopment analysis. Management Science, 30, 1078 1092. Carac, J., Conceic, P., Heitor, M.V., 2000. Towards a public policy for the research university in Portugal. Higher Education Policy, 13, 181 201. Carotenuto, G., Lapegna, M., Zollo, G., Di Donato, A., Nicolais, L., 2001. Evaluating research performance: the strategy of the University of Naples Federico II (Italy). Higher Education Policy, 14, 75 90. Charnes, A., Cooper, W.W., Rhodes, E., 1978. Measuring the efficiency of decision making unites. European Journal of Operation Research, 2(6), 429 444. Coelli, T., 1996. Assessing the performance of Australian universities using data envelopment analysis. Centre for efficiency and productivity analysis, University of New England, NSW. Dyer, J.S., 1990. Remarks on the analytic hierarchy process. Management Science, 36(3), 249 258. Johnes, G., 1988. Research performance indications in the university sector. Higher Education Quarterly, 42(1), 54 71. Johnes, G., Johnes, J., 1993. Measuring the research performance of UK economics departments: an application of data envelopment analysis. Oxford Economic Papers, 45, 332 347. Lindsay, A.W., 1982. Institutional performance in higher education: the efficiency dimension. Review of Educational Research, 52, 2, 175 199. Lowry, R.C., 2001. The effects of state political interests and campus outputs on public university revenues. Economics of Education Review, 20, 105 119. Mansfield, E., 1998. Academic research and industrial innovation: an update of empirical findings. Research Policy, 26, 773 776. Miyata, Y., 2000. An empirical analysis of innovative activity of universities in the United States. Technovation, 20, 413 425. Saaty, T.L., 1980. The Analytic Hierarchy Process. McGraw-Hill International Book Company. Stern, Z.S., Mehrez, A., Barboy, A., 1994. Academic departments efficiency via DEA. Computers and Operations Research, 21(5), 543 556. Tomkins, C., Green, R., 1988. An experiment in the use of data envelopment analysis for evaluating the efficiency of UK university departments of accounting. Financial Accountability and Management, 4(2), 147 164. Webster, T.J., 2001. A principal component analysis of the U.S. News & World Report tier rankings of colleges and universities. Economics of Education Review, 20, 235 244. Xu, H., Wang, H., 1998. Application of multivariate analysis in studying the graduate school s scales of universities in China. Journal of Beijing University of Aeronautics and Astronautics, 24(2), 245 248.