An AHP/DEA method for measurement of the efficiency of R&D management activities in universities

Transcription

1 Intl. Trans. in Op. Res. 11 (2004) INTERNATIONAL TRANSACTIONS IN OPERATIONAL RESEARCH An AHP/DEA method for measurement of the efficiency of R&D management activities in universities Y.J. Feng a,h.lu a and K. Bi b a Management School of Harbin Institute of Technology, Harbin and b Science & Technology Management Institute of Harbin University of Science and Technology, Harbin , China fengyi@hope.hit.edu.cn [Feng]; lvhui@263.net [Lu] Received 25 September 2002; received in revised form 6 April 2003; accepted 30 June 2003 Abstract In order to develop a better tool for the assessment of the management performance of research and development (R&D) activities in research-oriented universities, a combination of analysis hierarchical process (AHP) and data envelopment analysis (DEA) is proposed for the assessment of the efficiency of R&D management activities in universities. The measure consists of the measurement of a university s previous and present R&D strength by AHP and the assessment of the relative efficiency of its growth in R&D strength against those of other universities by DEA, in which the management basis of the measured universities is taken into consideration. The application of the measure to assess the R&D management efficiency of 29 universities in China indicates the universities which have improved their management work achieved a high efficiency value regardless of whether their original R&D strengths were strong or weak. Such a measure is proved to be helpful for motivating the universities to keep on improving their R&D management. Keywords: Data envelopment analysis; management efficiency; R&D activities; analysis hierarchical process; ranking universities Introduction After the education reform, many colleges in China have expanded into comprehensive universities with departments across a wide variety of fields to further enhance their research and development (R&D) strengths. So the universities have to face up to the challenge of enhancing their management performance in R&D activities. As the assessment of the inner management work of R&D activities in universities helps to motivate the managerial staff of the universities to improve their work and then to enhance the R&D strengths of universities at large, r 2004 International Federation of Operational Research Societies. Published by Blackwell Publishing Ltd.

2 182 Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) a practical and reasonable measure is developed for the assessment of the efficiency of the management work at R&D activities in universities in this paper. The analysis hierarchical process (AHP), a comprehensive strength-evaluation method fails to measure the performance of management work in many cases because it neglects the influence of the diversity in management basis of the measured units on the evaluation result, which makes the results less efficient and objective. The same problem exists with the data envelopment analysis (DEA) method, a relative efficiency evaluation method of a group of units. In this paper, we propose a two-stage measure which combines the AHP and DEA to assess the management efficiency of R&D activities in universities. In the first stage, a university s previous and present R&D strength is measured by AHP and the relative efficiency of its growth in R&D strength against those of other universities is assessed by DEA. Thus the efficiency of R&D management activities in a university is measured by its growth rate in R&D strength according to other measured universities. In this way, the measure overcomes the shortcoming of AHP and DEA itself when used for assessing the performance of management work. Such a measure is helpful for motivating the measured universities to improve their inner management work at R&D activities and to explore efficiency ways to enhance their R&D strengths. This paper is organized as follows. In the second section we will review the evaluation work of universities. The mathematical model for our proposed measure is presented in the third section. In the fourth section, we apply the measure to assess the efficiency of R&D management activities in 29 universities in China to give a further illustration of the method. Literature review Since the US news and World Report published its ranking result for US universities for the first time in 1983, ranking activities for universities have become popular worldwide and have, to a great extent, promoted the development of universities (Webster, 2001). With the transference of a university from the center of education to centers of education and R&D (Mansfield, 1998), rating activities of universities have extended from the assessment for teaching quality to comprehensive strength, in which R&D standing takes an increasing heavy weight (Ahn, Charnes and Cooper, 1998; Johnes, 1988). Factors contributing to the success of R&D activities in universities include policy tendency (Lowry, 2001; Carac, Conceic and Heitor, 2000), good organization arrangement (Lindsay, 1982), pursuit of a good reputation for the university authority (Baden-Fuller, Ravazzolo and Schweizer, 2000), inner innovation motivation in universities (Miyata, 2000). However, the key factor which will bring out an increase in strength is the efficient inner management of R&D activities. Many methods are used for the assessment of the R&D activities in universities, among them the analysis hierarchical process (AHP), factor analysis and cluster analysis are also very popular (Xu and Wang, 1998). These provide a description of the strength standing of the measured universities. However, for the purpose of analyzing the management work at R&D activities in universities, efficiency analysis is usually a practical way. The DEA method provides a feasible and simple way for relative efficiency evaluation of public sectors characterized by multi-input and multi-output (Charnes, Cooper and Rhodes, 1978) and is widely used for the assessment of

3 Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) universities, from research efficiency of universities (Ahn, Arnold, Charnes and Cooper, 1989; Avkiran, 2001; Coelli, 1996) and teaching efficiency of departments ( Johnes and Johnes, 1993; Tomkins and Green, 1988) to scaling graduate school (Stern, Mehrez and Barboy, 1994). However, few of them focus on the assessment of performance of management work in universities, which will provide an insight into the potential growth in R&D strength of universities, and will be helpful for enhancement of R&D strength from a strategic development view. For this purpose, we present our measure for the efficiency of management work. Model The first stage: AHP method AHP, developed by Satty in 1980 (Saaty, 1980), is a simple and feasible multi-objective evaluation method widely used for multi-object evaluation activities. It is designed for subjective evaluation of a set of alternatives based on multiple criteria organized in a hierarchical structure. At the top level, the criteria are evaluated and at the lower levels, the alternatives are evaluated by each criterion. The decision-makers assess his evaluation separately for each level and sub-level subjectively. By creating a pairwise comparison matrix, his subjective evaluation for every pair of items is assessed (Dyer, 1990). In the first stage of our measure, by AHP, we get the weight of each indicator of a hierarchical indicator system established for the assessment of the R&D strength of a group of researchoriented universities. The weighted sum of all indicator data of a university is treated as its strength in the data time. Here, we use the common title decision-making unit (DMU) for a measured university. Then we give the following definitions: Definition 1: the historical or previous strength of a DMU is its Reference Index (RI) Definition 2: the present or later strength of a DMU is Present Index (PI) Definition 3: the RI and PI constitutes the Index State of a DMU, represent by (RI, PI) The second stage: DEA method Suppose there are n DMUs. DMU j represents the jth DMU. x j represents the RI of DMU j and y j represents the PI of DMU j. Then (x j,y j ) represents the IS of DMU j. We define T as the Index Possibility Set made up by the DMUs. The definition of T is shown in (1). ( ) T ¼ ðx; yþj Xn l j x j 4x; Xn l j y j 4y; Xn l j ¼ 1; l j X0; j ¼ 0; 1; 2; :::; n ð1þ j¼0 j¼0 j¼0 wherein ðx 0 ; y 0 Þ¼ð0; 0Þ. Obviously, T is a convex set, viz. if ðx 0 ; y 0 Þ2T, ðx 00 ; y 00 Þ2T, then ðlx 0 þð1 lþx 00 ; ly 0 þð1 lþy 00 Þ2T; 04l41.

4 184 Using the following single input to single output DEA model presented by Banker, Charnes and Cooper (1984) as shown in (2) on T, we can get the relative efficiency of each DMU. max Z s:t: Xn l j a j 4x j0 X n j¼0 X n j¼0 j¼0 l j y j Xzy j0 l j ¼ 1 l j X0; j ¼ 0; 1; 2; :::; n Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) wherein x j is the RI of DMU j and y j is the PI of DMU j, j ¼ 0; 1; 2:::; n, x 0 ¼ 0; y 0 ¼ 0. Suppose Z 0 is the optimal value of DMU j by (2). Set Z j the management efficiency (ME) of DMU j, then Z j ¼ 1=Z 0 100% ð3þ If Z 0 5 1, the corresponding DMU is on the frontier of T. Suppose z 0 is the optimal value of ðx j0 ; y j0 Þ by (2), set x j0 ¼ x j0 ; y j0 ¼ z 0 y j0, it can be seen that ðx j0 ; y j0 Þ is right on the frontier of T, as shown in Fig. 1. We refer ðx j0 ; y j0 Þ as the projection point of ðx j0 ; y j0 Þ on the frontier of T. Then the ME of ðx j0 ; y j0 Þ is Z j0 ¼ y j0 =y j0 100% ¼ 1=Z 0 100% ð2þ Illustration of the model The principle of the above measure can be illustrated by Fig. 2. In Fig. 2, A, B, and C are three DMUs respectively. Their corresponding RIs are along the X axis as input and their PIs are along the Y axis taken as output. Suppose A and C are two DMUs whose growth in strength is optimal at its standing. As the RI of B is higher than that of A but lower than that of C, as shown in Fig. 2, B lies between A and C. B is under the linked line from A ( x j0, y j0 ) C D Y : Present Index A B ( x, y j0 T j0 ) 0 X : Reference Index Fig. 1. Illustration of the model.

5 Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) Y: Present Index A ( x A, y A) B ( x B, yb ) B( x B, yb C x C, y ) ( C ) 0 xa xb X : Reference Index x C to C. This means the efficiency of the management work in B is less efficient as it did not enhance its strength enough to reach the optimal level. In other words, if B is efficient as A or C, it should reach to the strength level line from A to C with its RI. Therefore B is inefficient in the measure and its efficiency value is measured by the ratio of its PI to the corresponding point in line AC, B 0 in this example. Measurement of the management work of R&D Activities at 29 universities in China Evaluation indicator system and data A three-tier indicator system is established for the evaluation of comprehensive R&D strength of universities in China, in which the R&D input and R&D output are considered together. The R&D input of a university is measured by its personnel structure and R&D expense. And the R&D output is usually measured by the number of published papers, number of research and applied projects. Moreover, patents and identified R&D projects are also important for the assessment of the R&D standing of a university. Using a questionnaire from some experts from the Education Appraisal Institute affiliated to the National Education Ministry of China, we get the relative weight for each indicator in the evaluation index system through AHP, as shown in Table 1. The universities being measured in our example are 29 universities which are affiliated to the National Education Ministry of China and the indicator data has been selected from the Chinese Higher Education Statistical Data. The weighted sum of all indicator data of a university represents its R&D strength standing in the data time. In this case, we use the weighted sum of all indicator data of a university in 1996 as its RI (namely, its management basis) and that of it in 1997 as its PI (its performance of management work in 1997). In this way, we can get the RIs and PIs for all measured universities, shown in Table 2. Then using the DEA model above, with the RIs of the 29 universities as input and the PIs of them as output, we can get their MEs in 1997, as shown in Table 3. Result and analysis Fig. 2. Illustration of the principle of the measure. Using RIs of all measured universities as input X and PIs of all measured universities as output Y, we can produce the graph in Fig. 3, which provides an insight into our measure.

6 186 Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) Table 1 Indicator system for evaluating the R&D management efficiency of universities in China First-tier indicator Second-tier indicator Third-tier indicator Weight Full-time R&D personnel Number of academic faculty Professors 1.84 Professors 1.22 Lecturers 0.55 Teaching assistants 0.36 Others 0.17 Number of other faculty Senior faculty 1.34 Junior faculty 0.54 Primary faculty 0.19 R&D expense 6.21 Project R&D project Basis research projects 2.95 Fruits of science and technology Application research 1.10 projects Experimental and 0.61 developmental projects Technology service project Application of other R&D projects Publication International 7.45 publications National publications 1.76 Regional publications 0.69 Monograph 0.77 Identified product International level 2.75 National initiation 1.16 National leading 0.63 Others 0.27 Patent Inventions 3.7 New practical 0.79 application Appearance design International communications People attending international conference People attending 3.51 conference held at home Visiting scholar 1.76 Postgraduate abroad 0.87 Awards National awards 24.9 Provincial or departmental 9.05 awards Regional awards 3.28 Income of technology transference The origin (O) and the four optimal universities, North-east Normal University (5.8, 12.68), Fudan University (28.53, 41.48), Peking University (50.51, 51.02), and Zhejiang University (64.9, 56.7) construct the optimal management frontier ONFPZ of our case. In the measure the ME of

7 Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) Table 2 Reference indexes and present indexes of 29 universities in China. University RI (Comprehensive strength standing of 1996) PI (Comprehensive strength standing of 1997) RI Order PI Order Peking University Beijing Normal University Nankai University Tianjing University Dalian Science and Engineering University Jilin University North-Eastern Normal University Fudan University Tongji University Shanghai JiaoTong University East China University of Science and Technology East China Normal University Nanjing University South-East University Zhejiang University Xiamen University Shangdong University Qingdao Sea University Wuhan University Huazhong University of Science and Technology Central China Normal University Zhongshan University South China University of Science and Technology Sichuan Union University Chongqing University South-West Normal University Xi an Jiaotong University Shanxi Normal University Lanzhou University a university in the group is measured by the ratio of its PI to its corresponding projection point on the frontier ONFPZ. Take Fudan University, Tongji University, and Xi an Jiaotong University ( points F, T, and X in Fig. 3) for example. It is obvious that the ME of Fudan University is higher than that of Tongji because with very close R&D strengths in 1996 (RIs in our measure), Fudan achieved in 1997, much higher than Tongji s The situation is quite different when comparing Fudan to Xi an Jiaotong University. Although the strength of Xi an Jiaotong University in 1997 is a little higher than that of Fudan, 41.6 to 41.48, the ME of Xi an Jiaotong is only 93.41, lower than Fudan s 100 because with quite better original strength standing, in 1996, Xi an Jiaoting s growth in strength is much smaller. This means the management work in Xi an Jiaotong

8 188 Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) Table 3 R&D management efficiency of 29 universities in China in The measured universities Strength in 1997( PI) Management efficiency PI Order ME Order Peking University Beijing Normal University Nankai University Tianjing University Dalian Science and Engineering University Jilin University North-Eastern Normal University Fudan University Tongji University Shanghai JiaoTong University East China University of Science and Technology East China Normal University Nanjing University South-East University Zhejiang University Xiamen University Shangdong University Qingdao Sea University Wuhan University Huazhong University of Science and Technology Central China Normal University Zhongshan University South China University of Science and Technology Sichuan Union University Chongqing University South-West Normal University Xi an Jiaotong University Shanxi Normal University Lanzhou University Fig. 3. Illustration of management efficiency of 29 universities in China.

9 Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) Table 4 Data of indicator of Beijing Normal University, Northeast Normal University and East China Normal University. Indicator Beijing Normal University (9.95, 11.98) North-east Normal University (5.8, 12.68) Lanzhou University (10.65, 13.71) Weight Growth Rate (%) Growth Rate (%) Growth Rate (%) National awards F Technology transfer Departmental awards International publications R&D expense People attending international conference Invention patent People attending conference held at home Regional awards F Basic research Identified project of international level Professor National publications Visitor scholar Senior faculty Associate professor R&D fruit Fruit of national initiation Applied research

10 190 University is less efficient. Or in other words, under the same management, the R&D strength of Xi an Jiaotong University in 1997 should have achieved rather than Using Northeast Normal University (N in Fig. 3) as another example, the comprehensive strength of N in 1997 is far lower than Fudan University, to F s But its R&D strength has a remarkable increase, from 5.8 in 1996 to in Such a growth rate is much higher than those of other universities with similar R&D strengths in 1996, for example Lanzhou University, L (10.65, 13.71) and Beijing Normal University, B (9.95, 11.98). Therefore, the ME of Northeast Normal University is as high as 100. Table 4 lists data of some key indicators (which are with weight more than 1) of the universities. From Table 4, we can see that some important indicators of Northeast Normal University generally achieved a higher increase from 1996 to Some indicators even get a breakthrough from zero. Although the important indicator data of Beijing Normal University and Lanzhou University has also increased to some extent, their growths in strength are far lower than Northeast Normal University. Thus their MEs are 66 and 78 respectively. In the measure, the efficiency of a university is assessed by its corresponding criteria defined by the behavior of a group of universities. Thus the measure fulfills our goal to assess the management work with full consideration of the original standing of the measured universities. By discussion of the result with some experts from the Education Appraisal Institute, their reflection on the result is positive. Most of them hold the idea that the result is reasonable and can provide much information about the inner management work at R&D activities in the measured universities. They agree that the measure can dig out more useful information which is helpful for the improvement of their management work and motivate the university to keep on improving their work. Conclusion As the assessment of the inner management work at R&D activities in universities is helpful for motivating the managerial staff of the universities to improve their work from a long strategic development view, a two-stage method which combines the existing methods of AHP and DEA method is developed in this paper. In the measure, the original R&D strength of a university is taken as its management basis for the assessment of its management work and its growth rate in R&D strength against those of other universities is taken as its efficiency of management work. The application of the measure to assess the efficiency of R&D management work of 29 researchoriented universities in China shows the efficiency values of universities varies a lot from their present R&D strengths. The universities with high efficiency value in the case improved their work a lot, which resulted in a higher increase in their R&D strength than those of other universities. The measure is proved to be reasonable and practical for the assessment of management work and it can also provide insight into the evolution of the R&D management work in universities when it is used for a long period of time. Acknowledgements Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) This paper is taken from the project Study on Appraisal Theory and Methods of Performance Management (No ). Supported by the National Science Foundation of China.

11 References Y.J. Feng, H. Lu and K. Bi / Intl. Trans. in Op. Res. 11 (2004) Ahn, T., Charnes, A., Cooper, W.W., Some statistical and DEA evaluations of relative efficiencies of public and private institutions of higher learning. Socio-Economic Planning Sciences, 22(6), Ahn, T., Arnold, V., Charnes, A., Cooper, W.W., DEA and ratio efficiency analyses for public institutions of higher learning in Texas. Research in Governmental and Nonprofit Accounting, 5, Avkiran, N.K., Investigating technical and scale efficiencies of Australian universities through data envelopment analysis. Socio-Economic Planning Sciences, 35, Baden-Fuller, C., Ravazzolo, F., Schweizer, T., Making and measuring reputations: the research ranking of european business schools. Long Range Planning, 33, Ball, D.F., Quality measurement as a basis for resource allocation: research assessment exercises in United Kingdom universities. R & D Management, 27(3), Banker, R.D., Charnes, A., Cooper, W.W., Some models for estimation technical and scale efficiencies in data envelopment analysis. Management Science, 30, Carac, J., Conceic, P., Heitor, M.V., Towards a public policy for the research university in Portugal. Higher Education Policy, 13, Carotenuto, G., Lapegna, M., Zollo, G., Di Donato, A., Nicolais, L., Evaluating research performance: the strategy of the University of Naples Federico II (Italy). Higher Education Policy, 14, Charnes, A., Cooper, W.W., Rhodes, E., Measuring the efficiency of decision making unites. European Journal of Operation Research, 2(6), Coelli, T., Assessing the performance of Australian universities using data envelopment analysis. Centre for efficiency and productivity analysis, University of New England, NSW. Dyer, J.S., Remarks on the analytic hierarchy process. Management Science, 36(3), Johnes, G., Research performance indications in the university sector. Higher Education Quarterly, 42(1), Johnes, G., Johnes, J., Measuring the research performance of UK economics departments: an application of data envelopment analysis. Oxford Economic Papers, 45, Lindsay, A.W., Institutional performance in higher education: the efficiency dimension. Review of Educational Research, 52, 2, Lowry, R.C., The effects of state political interests and campus outputs on public university revenues. Economics of Education Review, 20, Mansfield, E., Academic research and industrial innovation: an update of empirical findings. Research Policy, 26, Miyata, Y., An empirical analysis of innovative activity of universities in the United States. Technovation, 20, Saaty, T.L., The Analytic Hierarchy Process. McGraw-Hill International Book Company. Stern, Z.S., Mehrez, A., Barboy, A., Academic departments efficiency via DEA. Computers and Operations Research, 21(5), Tomkins, C., Green, R., An experiment in the use of data envelopment analysis for evaluating the efficiency of UK university departments of accounting. Financial Accountability and Management, 4(2), Webster, T.J., A principal component analysis of the U.S. News & World Report tier rankings of colleges and universities. Economics of Education Review, 20, Xu, H., Wang, H., Application of multivariate analysis in studying the graduate school s scales of universities in China. Journal of Beijing University of Aeronautics and Astronautics, 24(2),

12