Do University Units Differ in Efficiency of Resource Utilization?

Transcription

1 CESIS Electronic Working Paper Series Paper No. 176 Do University Units Differ in Efficiency of Resource Utilization? Zara Daghbashyan December 2012 The Royal Institute of technology Centre of Excellence for Science and Innovation Studies (CESIS)

2 Do University Units Differ in Efficiency of Resource Utilization? Case Study for the Royal Institute of Technology (KTH), Stockholm 1 Zara Daghbashyan 2 Abstract The efficiency of universities is attracting increased interest, with most studies comparing the performance between different universities. However, the within-university variation is largely overlooked in the literature. Using data envelopment analysis this paper identifies heterogeneity in the performance of 47 units of a leading Swedish university (the Royal Institute of Technology) in terms of resource utilization. The findings suggest the following: First, three quarters of the units exhibit similar high performance. Second, the units are more efficient in using resources for research than for teaching. Third, efficiency in research is highly correlated with efficiency in teaching, implying a complementary relationship. JEL classification: C14, I21, I23 Keywords: Technical and scale efficiency, data envelopment analysis, universities 1 I am grateful to Professor Gary D. Ferrier, University of Arkansas, for the valuable comments and review of the paper at EWEPA 2009 conference in Pisa, Italy. 2 Industrial Economics and Management, Royal Institute of Technology, Stockholm; zarad@kth.se 2

3 1. Introduction In today s knowledge economy more and more attention is being paid to the higher education sector, the producer of human capital and knowledge. Aspiring to contribute to the production of human capital and knowledge, which is deemed the driving force of economic growth (Romer, 1986, Lucas, 1988), governments allocate public financing to higher education institutions. The latter use the funding to perform their main functions, teaching and research as well as to realise their third mission, i.e. dissemination of knowledge and interaction with the society. However, while implementing the same functions and producing relatively similar outcomes, the higher education institutions, namely research universities, may exhibit differences in efficiency of operation. Economic theories of non-profit behavior postulate that organizations like higher education institutions have little incentive to engage in efficient production practices (James, 1990). Niskanen (1971) suggests that public organizations are budget maximizers and hence have greater freedom to pursue their own objectives at the expense of conventional objectives (in Kumbhakar and Lovell, 2000). As noted in Robst (2001) the activity of higher education institutions might be driven by the pursuit of excellence and prestige maximization, which does not necessarily imply economic efficiency traditionally assumed for profitmaximizing business establishments. Nonetheless, due to budget restrictions, public authorities allocating funds to universities are interested not only in the excellence and prestige of universities, but also in the efficient utilization of their resources. Universities, being less restricted in their use of resources might be reluctant to operate efficiently, with irrational use of external finances as a result. In this respect the measurement of university performance in terms of resource utilization becomes an important issue. Do universities differ in the efficiency of resource utilization or do they exhibit similar performance while pursuing the same prestige maximization objective? This is important not only for those allocating financing to universities, but also for the universities themselves. They can utilize knowledge of their relative efficiency to eliminate existing shortcomings and show higher performance. Several studies have been conducted to evaluate the technical efficiency of higher education institutions. Most of them focus on estimating the technical efficiency of a group of universities or university departments in a given country. The average technical efficiency reported in those studies seems to range from 80 to 95 percent. Johnes (2006a) evaluated the performance of 100 higher education institutions in England and found that the mean technical efficiency of English universities is 93 percent compared to the best in the sample. 3

4 Agasisti and Salerno (2007) estimated technical efficiency of Italian universities and reported an average efficiency of about 84 percent. Similar results were achieved by Abbot and Doucouliagos (2003), who analyzed the technical efficiency of Australian universities. However, an efficiency study of Spanish universities by Garcia-Aracil & Palomares-Montero (2008) revealed that the average technical efficiency of Spanish public universities was below 70 percent relative to the best performing one. The technical efficiency analysis conducted for 30 Swedish higher education institutions (HEI) suggests that that about half of Swedish HEIs exhibit similar performance with average technical efficiency of 93 percent (Riksrevisionen, 2011:2). The majority of university efficiency studies focus on the variation in efficiency among universities of the same country. Assessments have been made of the overall performance of universities compared to the best performing ones. However, the possibility of efficiency variation within the same university is seldom taken into account. Goldstein & Thomas (1996) have shown that both within and between institutional variations are important for constructing a measure of performance. Accordingly, it is important to know the extent to which the average efficiency score of the university reflects the performance of units within the same university. Could it be that, despite operating in more or less similar conditions, and having the same administration and similar strategic approaches, they exhibit different performance and efficiency? There are a few studies that measure the technical efficiency of university departments, among them Kao et. al (2008) and Tauer et. al (2007). They find that there is a variation in the efficiency among university departments, i.e. university units differ in their ability to make optimal use of resources. Tseremes and Halkos (2010) studied the efficiency of 16 institutions of the same university in Greece and found that the average efficiency was 85 percent. The findings from these studies imply that the average efficiency of a university does not necessarily describe the operation of all university units. This paper uses the Royal Institute of Technology (KTH, Stockholm) as a case study to shed more light on the variation in the technical efficiency within the same educational organization. It contributes to the existing literature by suggesting a methodology for comparing the relative performance of university units and by using a unique dataset which allows the quality of university research to be controlled for. An added contribution is the analysis of the mutual relationship of the efficiencies in performing two major university functions, teaching and research. Some units may be efficient in utilizing their resources for teaching purposes; others may have high efficiency in research production, while a third 4

5 group may be efficient in both or inefficient in both dimensions. The estimations will hence make it possible to see if research and education are positively or negatively related in terms of efficiency. Thus, the relative performance of 47 units, operating within KTH and producing technical education and research, is measured using data on teaching and research outputs of the university for the year It is worth noting that KTH accounts for one-third of Sweden s technical research and engineering education capacity at the university level. There are just over 12,000 full-year equivalent undergraduate students, more than 1,400 active postgraduate students and 2,800 full-time equivalent employees. Every year the university awards about 2000 bachelor and master degrees and 400 licentiate and PhD degrees 4. Data envelopment analysis (DEA) is used to measure the technical efficiency of KTH units in teaching and research both separately and jointly. The rich dataset, from the Research Assessment Exercise (RAE) 5 conducted by KTH, allows the analysis to cope with difficulties in measuring research output mentioned in other studies. In particular, detailed information on research outputs is utilized to control for both output heterogeneity and quality. According to the results 6 about three quarters of KTH units operate on the frontier, i.e. they have similar high efficiency in utilization of their resources for teaching and research activities. The average performance of inefficient units is estimated to be 83 percent. The inefficient operation seems to be the result of both poor management and scale inefficiency. The results also indicate that the units that are technically efficient in either research or teaching have more chances to be technically efficient in their activity as a whole. There is a strong positive correlation between the efficiency of resource utilization in teaching and research, suggesting that efficiency in research affects the efficiency in teaching and vice versa. The paper is organized as follows: the concept of technical efficiency is presented in Section 2. DEA is described in Section 3, input-output indicators and data sources are discussed in Section 4. The analysis and results are presented in Section 5. Robustness check and the summary of results are in Section 6 and 7 respectively. 3 The data were kindly provided by KTH administration. 4 See for more details 5 See The International RAE Project Report by KTH University Administration available 6 The results of this analysis have been used by KTH administration for comparison of resource utilization across KTH units. 5

6 2. Technical efficiency and production frontiers Classical economic theory assumes that firms seek to maximize profit or minimize cost and thus operate efficiently, but evidence from practice does not always support it. Some firms, especially those operating as non-profit organizations, tend to deviate from the predicted behavior and are hence regarded as inefficient (James, 1990). James and Neuberger (1981) argue that this happens due to different optimization strategies. In contrast to firms and organizations operating for profit, non-profit organizations have other objectives 7 and profit maximization, usually assumed for efficient operation, does not seem to be is a reasonable goal for them. Another line of behavioral theory argues that whatever objectives non-profits pursue, they are inherently subject to productive inefficiency due to absence of ownership claims to residual earnings or profits, due to relatively slow response to demand changes or additionally due to specifics of income generating processes (Hansmann, 1996). The pioneering work by (Farrel, 1957) provided the definition and conceptual framework for analyzing efficiency. Farrel suggested that efficiency can be usefully analyzed in terms of realized deviations from an idealized frontier. Kumbhakar and Lovell (2000) define the production frontier as the boundary of the graph of the production technology. This boundary represents the maximum output that can be produced given inputs or, alternatively, minimum inputs required to produce given output. That is, it presents the best practice performance in the industry/organization. Efficiency measures are defined in such a way as to provide measures of distance to a respective frontier function. The traditional definition of technical efficiency concerns the efficiency of resource utilization; that is, how efficiently inputs are transformed into outputs compared to the best performing unit. A producer is technically efficient if, and only if, it is impossible to produce more of any output without producing less of some other output or using more of some input. The basic concept of technical efficiency is illustrated in Figure 1. The axes show the inputs (x 1 and x 2 ) per unit of output (q). SS' represents all input-output combinations possible when the available technology is efficiently used, i.e. the frontier production. The operation of any decision-making unit (DMU) can be represented by a point above or on SS'. Point P represents a particular DMU. Since P is not on SS', it is not efficient because inputs per unit of output exceed the technically possible. The technical efficiency can be increased by 7 For example instead of maximizing profits they aim to maximize the quality and quantity of services produced (James and Neuberger, 1981). Another explanation is suggested by Niskanen (1971), who argues that non-profit organizations are budget maximizers because it enhances the apparent importance of the organization or alternatively provides the preferred trade-off between quantity and quality maximization 6

7 proportional reduction of inputs along OP to the point Q on the frontier. It is reflected in the distance QP and measured as OQ/OP (Coelli et al., 2005). Figure 1. Production Frontier and Distance Functions Two methods that are most often used to construct frontiers are data envelopment analysis (DEA) and stochastic frontier analysis (SFA), which involve mathematical programming and econometric analysis, respectively (Bogetoft et al., 2011). In SFA the frontier production function is constructed using regression analysis and all DMUs are assumed to have the same production technology. DEA, in contrast, optimizes the performance measure of each DMU. The focus is on the individual observation and n optimization problems (one for each DMU) are solved to construct the frontier (Kumbhakar and Lovell, 2000). Under SFA assumptions, deviations from the frontier are separated into two components, i.e. inefficiency and random noise. The cost paid for the separation of inefficiency components from random noise is the imposition of a specific functional form for the regression equation (production or cost function) relating independent variables to dependent variables. Furthermore, some specific assumptions about the distribution of the random error term are necessary, which in practice are not theoretically grounded. Thus, DEA is a deterministic method with a basic assumption that all deviations from the frontier production function are solely due to inefficiencies. This approach is limited when it comes to utilization of statistical tests, but its obvious advantage is that it does not require any assumptions about the functional form 8. The maximal performance measure for each DMU is calculated relative to all other DMUs in the observed population with the sole requirement that each DMU lies on or above the frontier (Fig.1). Furthermore, DEA allows for models with multiple outputs, which is hard to accomplish using SFA. This is achieved due to the 8 except for convexity assumption 7

8 possibility of assigning optimal weights to the outputs, whereas SFA models mainly allow for one output in the form of a dependent variable. Both approaches have advantages and disadvantages and the choice of the method depends on the specific situations. DEA has been used extensively for efficiency analysis in public sector organizations, which are often characterized by multiple inputs and outputs with no price information. Many authors use DEA as a performance measurement tool in the education sector (see Abbot & Doucouliagos, 2003, Johnes, 2006a, Fare et al.,1989), and argue that DEA is a more appropriate tool for multi-output production of educational establishments. In this paper the technical efficiency of 47 units of KTH is assessed with the DEA method, due to the possibility of using multiple outputs and avoiding functional form assumptions about the production function and random noise distribution. 3. The DEA Method There are two possible approaches to DEA problems, input-oriented and output-oriented (see more in Coelli et al., 2005, Bogetoft et al., 2011). The input-oriented approach considers the option of reducing inputs for the given value of outputs, while the output-orientation approach deals with the expansion of outputs with given inputs. The explanations provided below refer to the output-oriented approach, which in our opinion is more appropriate for this analysis. This is because the inputs to the university units operation tend to be fixed in the short-run and the units are more flexible in their choice of outputs. To provide some insight into the DEA method, we start from its simple form. Consider n DMUs each using m inputs to produce s outputs. Assume each DMU aims to optimize its productivity, i.e. to maximize the ratio of outputs to inputs or, equivalently, minimize the ratio of inputs to outputs. In conditions of multiple inputs and outputs, some weights should be assigned to outputs and inputs, but there are no commonly agreed weights that could be used. Prices may be used as a means to aggregate different output and input categories, but there is no price information available for the output of public establishments. DEA suggests a method to optimize the ratio of weighted outputs to weighted inputs (or, equivalently, minimize the ratio of weighed inputs to outputs) through assigning optimal input and output weights for each DMU. It is assumed that all DMUs have equal access to inputs though they differ in amounts of inputs used and outputs produced. The optimal weights for each DMU depend on the amounts of inputs and outputs, and differ across DMUs. Let y o be the vector of outputs produced by DMUo and x o the vector of inputs used by DMUo. Each DMU chooses a vector of input and output weights, v and u respectively, so as to optimize the ratio of weighted inputs to outputs. 8

9 min h (u,v) o m i 1 s r 1 v x u i r y io ro (1.0) It is worth noting that the measurement units of the different inputs and outputs need not be congruent. Some may involve the number of people, or areas of floor space, or money expended etc. (Cooper et al., 2007). Each DMU is allowed to select the weights which are most favorable for optimizing (1.0), but without additional constraints (1.0) is unbounded. A set of normalizing constraints (1.2) reflects the condition that the weight vectors chosen by DMUs should not allow any DMU to achieve a ratio of weighted inputs to weighted outputs less than unity. Thus, the following fractional linear programming problem is formulated for every DMU min h (u,v) m o m i 1 s r 1 Subject to v x u i ij r rj i 1 r 1 s i r y io ro v x u y 1 (1.1) (1.2) for j=1, n u, v r i 0 (1.3) The above problem yields an infinite number of solutions, because if (u*, v*) is optimal, then (αu*, αv*) is also optimal for any positive α. Charnes and Cooper (1962) suggested a method to restrict the number of solutions by imposing a new constraint which requires the sum of weighted outputs to be equal to unity. As a result, the above fractional programming problem is transformed into the following equivalent linear programming problem: m min q vi x r 1 Subject to io (2.1) m v x u y 0 for j=1, n (2.2) i ij r rj i 1 r 1 s s uy r ro 1 (2.3) r 1 u, v 0 (2.4) r i The constraints (2.2) are transformed from (1.2) under the condition of (2.3). Solving the problem for all DMUs, the corresponding optimal weights, which maximize the virtual productivity, are found for each DMU. The technical efficiency score, θ, which is equal to 1/q, 9

10 is found by solving the dual to this linear programming problem. The dual problem aims at maximization of the technical efficiency score under the requirement that the efficiency corrected volume of output must not exceed the amount of output produced by the reference units, whereas the amount of inputs must at least equal the amount of inputs used by reference units. The DMUs with a technical efficiency score of unity are deemed technically efficient. The solution of the dual to the above described linear programming problem finds not only technical efficiency scores, but also input output slacks 9. It enables judgments on the change required in the input-output mix for both efficient and inefficient DMUs to be transformed into strongly 10 efficient DMUs. In addition, the dual problem also suggests a respective reference set for each inefficient DMU, identifying efficient DMUs whose activity each inefficient DMU should emulate to be on the frontier , known as CCR model, is built under the assumption that the frontier production technology exhibits constant returns to scale (CRS); it does not allow for the possibility of variable returns to scale (VRS), which is more likely in practice. To overcome this drawback, Banker-Charnes-Cooper (1984) further developed the model via inclusion of a free sign variable voin the objective function, which resulted in the following model known as outputoriented BCC model. m m min q v x v s r 1 Subject to i ij r rj o i 1 r 1 i io o (3.1) v x u y v 0 for j=1, n (3.2) s uy r ro 1 (3.3) r 1 u, v 0 r i (3.4) v o free in sign (3.5) This model is more flexible; the scale of production is determined by the data and can vary across units. This information is used to make judgments on the scale efficiency discussed in Appendix II. Following the literature (Abbot, 2003, Johnes, 2006a), the output-oriented BCC model is chosen for our analysis. 4. Inputs and outputs of university operation As mentioned in Johnes (2006a), the classification of inputs and outputs is a crucial step in DEA, because the method is sensitive to the number of inputs and outputs used in the analysis 9 See Appendix II 10 See Appendix II 10

11 and their specification. Hence, the precise specification of inputs and outputs and the ability to make them quantifiable constitute the major challenge for this type of analysis. The primary outputs of research universities are education and research, both of which should be treated as intangible goods with no explicit market value. There is no price system for education and research with which to measure the value of the output. It may be possible to use tuition fees as a measure of teaching output for education but as argued in Triplett and Bosworth (2004), the tuition fee is not a good measure since there is a wide variation in the proportion of education costs it covers; moreover this approach is impossible for public institutions with no tuition fee. The same is true regarding the research output; there is no price system or common standard with which to value the research output. One proxy that may be used to measure the research output is the research funding, but, as argued by some authors, it does not necessarily reflect the outcome and can be considered to be the input and not output of research. Hence, in conditions of no price information for university outputs, some other measures should be used Teaching output The common approach to measuring the teaching output of education sector institutions is through the total number of graduates, number of full year equivalent students or their average performance (e.g., Abbot and Doucouliagos, 2003, Izadi et al., 2002, Johnes, 2006, Stevens, 2005). Nonetheless, there are several problems associated with these measures. The total number of graduates reflects the output produced as a result of teaching for more than one year and hence is not a good proxy for annual performance. The number of full year equivalent students used as an indicator of teaching output does not reflect the quality of teaching. The measure of teaching output used in this study is the full-year equivalent performance in undergraduate and graduate education, which reflects both the quality and quantity of teaching output as well as two different teaching levels. It is calculated via multiplication of the number of full-time equivalent students by the ratio of sum of credits earned during one academic year to the sum of credits due. One may argue that the average performance is the result of both students individual abilities and teaching quality. The counterargument is that students of one university, in our case KTH, have relatively similar backgrounds and abilities, since they have to satisfy KTH admission criteria, and therefore may be assumed to be more or less homogenous in their backgrounds and abilities. 11

12 4.2. Research output The measurement of research output is more difficult because it has neither specific product form nor market value. Furthermore, research outputs may vary in their quality and it is important to account for the quality heterogeneity. Many studies measure the research output by bibliometric indicators such as the number of publications in scientific journals or the number of citations per publication (Sarafoglou et al.,1996, Johnes & Johnes, 1993). Another measure of research output traditionally used by many authors is the research funding (Abbot et al., 2003, McMillan et el., 2006). Cave et al. (1988) suggest that research funds reflect the market value of the research conducted and may therefore be considered as a proxy to the market price. Still, all these methods can be criticized: for instance one may argue that the number of publications does not reflect the whole research produced, since many scientific outcomes with quite significant value are not published in journals for different reasons, and counting the research output by the number of publications may neglect a quite considerable volume of output produced. Research funding is criticized for not reflecting the scientific value of the research output. Another argument is that the research funding may be deemed an input, and not an output, to the research production. To overcome the above mentioned difficulties, the research output of KTH units is measured from different angles. The research output of each unit is split up into two categories to allow inclusion of both published articles and conference papers. In addition, proxies for scientific and market values of the research output are used to control for the quality aspects. The data is available from the bibliometric study conducted within the scope of the KTH Research Assessment Exercise. In 2008 KTH initiated RAE to evaluate the research output of the university with ambition to use evaluation results for channeling resources to a high quality research environment. The evaluation was based on three operational aspects: an international expert review, a bibliometric analysis and a self-evaluation. Based on these three aspects, the scientific quality of the research and its applied quality were assessed. The scientific quality refers to originality of ideas and methods, scientific productivity, impact and prominence and the applied quality assessment indicates the applicability of research in industry or society. The quality indicators were translated into a scale of 0-5, where 5 represented quality of a world-leading standard. It is worth noting that the RAE quality indicators were evaluated using the research conducted in the preceding six years, and it is assumed that the quality of the research output did not change during that period. In this paper, 12

13 the abovementioned two quality indicators are used to proxy the scientific and applied values of the research produced. As an alternative measure of research output this paper uses data on research funding to incorporate information about the market value of the research, the justification being that the research funding represents the market value of the research produced, i.e. it measures the willingness of the external world to pay for the research and hence reflects its market value Inputs The definition and measurement of inputs to university operation are also associated with certain difficulties. Ideally, information on all major labor and capital inputs, split up into categories to control for their heterogeneity, must be included in the analysis. However, in practice, there is always lack of data on capital inputs and difficulties in controlling for the heterogeneity in labor inputs. For the purpose of our analysis, the labor inputs contributing to the teaching and research production are divided into two categories, i.e. the number of professors and other academic staff, which includes associate professors, assistant professors, docents, and researchers. Such a division allows us to take into consideration the different qualifications and skills in the main labor inputs. Data on administrative and technical staff is used to incorporate information on the contribution of capital inputs, and it is assumed that the number of technical-administrative staff is proportional to the capital inputs of units Data The specifications of all input and output categories used in this paper, as well as their descriptive statistics, are presented in the table below. The data are provided by the KTH administration and refer to the year It is worth noting that the input-output indicators below refer to 47 KTH units of KTH, formed within the framework of the RAE described earlier. Some of them correspond to organizational units, while others are more than one organizational unit. Table 1. Descriptive statistics of inputs and outputs Inputs Max Min 11 Average SD No of Professors (1) No of Technical Admin Staff (2) No of Academic Staff (3) Research Output No of Journal Papers (1) No of Conference Papers (2) Scientific Quality (3) Applied Quality (4) For some DMUs the volume of inputs and outputs is very small or 0 since the DMUs are not natural units. 13

14 Table 1. (cont.) Max Min Average SD External Research Funds in ths. SEK (5) Internal Research Funds in ths. SEK (6) Teaching output Full-year eq. Performance in grad. education (1) Full-year eq. performance in undergr. education (2) As shown by table 1, the units differ considerably in both outputs and inputs. One can see a huge variation in all input categories, implying that the units differ in the size. Interestingly, they differ not only in size but also type of activity. Some of them are more research-oriented and have high research output indicators, while others are more teaching-oriented, resulting in different combinations of teaching and research outputs, as illustrated in Figure 2. The figure shows the relationship between the total teaching output and the two main research output categories, journal and conference papers. It is obvious from the figure that there is almost no correlation between two main output categories. Figure 2. The relationship between teaching and research outputs 5. DEA analysis The package DEA-Solver is used to measure technical efficiency via output oriented variable returns to scale model. As mentioned earlier, output-orientation focuses on the amount by which outputs can be increased without additional inputs; which in our opinion is more appropriate for university units, since they are more flexible in altering their outputs than inputs. As the model with variable returns to scale assumption is more flexible and allows measurement of scale efficiency, it is the model of our choice. It is worth noting that the variable returns to scale assumption was previously used to measure the efficiency of universities in, for example, Johnes (2006) and Abbot & Docouliagos (2003). 14

15 Thus, 47 units belonging to the different schools of KTH 12 are combined, and DEA analysis is conducted for the entire pool of KTH units. The latter are assumed to be independent decision-making units. In reality the input decisions are based upon a dialogue with the school dean, and the units have relatively weak disposal of inputs. On the other hand, the units analyzed have a fuller control over their outputs, which is another reason for choosing an output-oriented DEA model. As mentioned earlier, the major drawback of DEA models is that the relative efficiency score achieved by each DMU can be sensitive to input-output specification. Furthermore, if a large number of inputs and outputs is defined relative to the number of decision-making units, then most, if not all, units will be measured as efficient. As mentioned in Tauer et al. (2007), with many outputs and inputs, more decision making units will be found to be unique in the production of at least one output or the use of one of the inputs. Hence it is important to test different model specifications for the robustness of results. In this paper different output specifications and their different combinations are used to control for the sensitivity of results to the number of output variables. All the models tested can be divided into three main groups. In the first group the focus is on measuring the efficiency in resource utilization for teaching purposes only. These models assume that university units cannot alter their research output, i.e. the amount of research they produce is fixed and efficiency improvement may be attained via a change in teaching outputs only. This group of models provides measurement of the efficiency of resource utilization in teaching, given research volumes. It should be noted that the assumption of a fixed volume of either inputs or outputs requires modification of the BCC model in order to accommodate their non-discretionary character. The Banker and Morey (1986) model, adapted for the case of non-discretionary variables, is used. The second group focuses on research efficiency under the assumptions that the teaching output of university units is fixed and that the efficiency can be improved by change of research output only. Finally the third group of models investigates the joint efficiency in teaching and research, assuming that units can improve their efficiency through change in both teaching and research outputs. The outcome when applying models of this group is the measurement of the efficiency of resource utilization for the joint activity of university units. Overall, 5 models that differ in the choice and aggregation of outputs have been tested in each group. All the tested models have the same input categories, i.e., the number of professors, 12 KTH is organized in 9 Schools. 15

16 academic staff and technical-administrative staff, but differ in the choice of outputs. We start with a restricted number of outputs and increase them gradually to see how the results change Technical efficiency in teaching Table 2 presents the results from the first group of models. These models aim at measuring technical efficiency in teaching, with the resources given and, as mentioned earlier, they assume that the research output of every unit is fixed and cannot be altered 13. Table 2. Technical efficiency of KTH units in teaching under different output specifications * Unit Number 3inputs 3inputs 3inputs 3inputs 3inputs 3outputs 4outputs 5outputs 6outputs 4outputs Following the agreement with KTH administration the names of units are kept anonymous. 16

17 Table 2. (cont.) Unit Number 3inputs 3inputs 3inputs 3inputs 3inputs 3outputs 4outputs 5outputs 6outputs 4outputs No of eff. units No of ineff. units Mean eff. score * Outputs in the 1 st model: (i) No of journal publications,(ii) no of conference papers, (iii) the sum of full-year equivalent performance in graduate and undergraduate education Outputs in the 2 nd model: (i) No of journal publications, (ii) no of conference papers, (iii) no of full-year equivalent performance in graduate education, (iv) no of full-year equivalent performance in undergraduate education Outputs in the 3 rd model: (i)no of journal publications, (ii) no of conference papers, (iii) the sum of full-year equivalent performance in graduate and undergraduate education, (iv) scientific quality indicator, (v) applied quality indicator Outputs in the 4th model: (i)no of journal publications, (ii) no of conference papers, (iii) no of full-year equivalent performance in graduate education, (iv) no of full-year equivalent performance in undergraduate education, (v) scientific quality indicator, (vi) applied quality indicator Outputs in the 5th model: (i) research funding, (ii) no of full-year equivalent performance in graduate education, (iii) no of full-year equivalent performance in undergraduate education, (iv) research funding Inputs in 1-5 models: (i) no of professors, (ii) no of academic staff, (iii) no of technical-administrative staff Technical efficiency scores corresponding to the model with three inputs and three outputs are presented in the first column. In this model the research output is measured by bibliometric indicators such as the number of journal publications and conference papers, and the teaching output is measured by full-year equivalent student performance in both graduate and undergraduate education. According to the results of this model 23 units are identified as efficient and 24 as inefficient 14. Though the average efficiency score is 0.7, the median score of inefficient units is only 0.3, suggesting relatively high inefficiency of resource utilization in teaching. The results of a slightly different model corresponding to a 3-input and 4-output case (teaching output is represented by two categories: the full-year equivalent student performance in undergraduate and graduate education) shown in in the second column are almost identical. Though the number of efficient and inefficient units is almost the same, the average efficiency score of 0.8 is higher. As shown in table 1 of Appendix I the correlation coefficient between technical efficiency scores of these two models is The efficiency score of 2 out of 24 inefficient units is rather high, implying that they are very close to being efficient. 17

18 The teaching output indicators included in these two models take into account both quantitative and qualitative aspects of education; meanwhile the research output indicators are quantitative only. To account for the quality of research, two qualitative indicators, measuring scientific and applied quality, are included. The results are shown in columns 3 and 4, and refer to two models that differ in the aggregation of teaching output only. The number of efficient units has now increased to 32 with average efficiency scores of 0.83 and 0.87, respectively. The correlation coefficient between the efficiency scores of the 3 rd and 4 th models is Some units, which are identified as inefficient in the first two models, have now become efficient, which is due to having high qualitative indicators in research. For instance, unit 11, which is deemed quite inefficient in the models without research quality indicators, turns into efficient when research quality indicators are included. This is due to having high qualitative indicators. In the first four models the research output is represented by quantitative and qualitative indicators, but, as mentioned earlier, the research output can be also measured by monetary units. The results when using internal and external research funding as a measure of research output, and full-year equivalent student performance in undergraduate and graduate education as teaching outputs, are shown in the fifth column of table 1. According to the results, 28 units are identified as efficient and 19 as inefficient. The average efficiency score is Some units, identified as efficient by the first four models, have turned into efficient and vice versa, which can be explained by imperfect correspondence of different research measures. The correlation coefficient between research funding and the number of journal and conference papers is 0.6. Monetary measurement certainly indicates the ability of a university unit to attract financial resources and perhaps also the value the external world attributes to its research, but the quantitative and qualitative indicators are probably better measures of the real outcome. Thus, the results of the first group of models suggest that 22 units remain efficient and 10 units inefficient under all specifications. The mixed results for the remaining 15 units have to do with their sensitivity to the model specification and should be treated with some caution. The average efficiency score varies from 0.75 to The median value for the inefficient units ranges from 0.30 to 0.50, suggesting that they can produce more teaching given the current level of research Technical efficiency in research The second group of models measures the technical efficiency of units in the utilization of resources for research purposes, assuming the given amount of teaching. The same five model 18

19 specifications are tested in this group with the difference that now teaching is held constant. The results are presented in table 3. Table 3. Technical efficiency of KTH units in research under different output specifications* 3inputs 3inputs 3inputs 3inputs 3inputs Unit Number 3outputs 4outputs 5outputs 6outputs 4outputs

20 Table 3. (cont.) Unit Number 3inputs 3inputs 3inputs 3inputs 3inputs 3outputs 4outputs 5outputs 6outputs 4outputs No of eff. units No of ineff. units Mean eff. score * The outputs in models 1-5 are the same as in table 2. Overall, 22 units are identified as efficient and 9 as inefficient in all 5 models. Some units have very low efficiency scores, suggesting inefficient use of resources for research. Interestingly, the units that are inefficient in research turn out to be inefficient in teaching as well (94percent of cases), implying that the inefficiency in research is not compensated by efficiency in teaching. The median technical efficiency score of inefficient units varies from 0.5 to 0.8, which is higher than the median efficiency score in teaching. This may mean that university units are more efficient in producing research than teaching. The average efficiency score varies from 0.75 to 0.95, which is again higher than the corresponding values for teaching. The comparison of models in this group shows that the inclusion of research quality indicators affect the efficiency scores of some units, implying that they are sensitive to the output specification Technical efficiency in teaching and research The models of the third group focus on the efficiency in both teaching and research, and reflect that inefficiency can be eliminated by altering the amounts of both research and teaching. As in the first groups, we start with models which include teaching and research quantitative indicators, and then extend them by the inclusion of research quality indicators. Following the logic of the previous sections, a model with monetary measures of research output is also tested, and the resulting five sets of technical efficiency scores are presented in table 4. Table 4. Technical efficiency of KTH units in teaching and research under different output specifications Unit Number 3inputs 3inputs 3inputs 3inputs 3inputs 3outputs 4outputs 5outputs 6outputs 4outputs

21 Table 4. (cont.) Unit Number 3inputs 3inputs 3inputs 3inputs 3inputs 3outputs 4outputs 5outputs 6outputs 4outputs No of eff. units No of ineff. units Mean eff. score * The outputs in models 1-5 are the same as in table 2. According to the results of the first two models, about 55 percent (26) of all the units are efficient; the third and fourth models suggest that 70 percent (33) of units are efficient. This is partly explained by the increased number of output indicators and the inclusion of research quality indicators. The fifth model, which differs from the first four in the specification of 21

22 research indicators, suggests efficiency of 57percent (28) of the units, implying that fewer units are efficient in utilizing their resources to attract research funds. Overall 26 units are identified as efficient and 11 as inefficient in all five models. The remaining 10 are sensitive to the model specification. The median technical efficiency score of inefficient units varies from 0.60 to 0.85, implying that they can increase their efficiency through altering outputs by percent Comparison of technical efficiency scores in teaching, research and teaching and research jointly The comparison of the technical efficiency scores from three groups shows that 21 units stay efficient and 9 inefficient under all specifications, implying that the efficiency of these units does not depend on the specification and the model chosen. These units show robust efficiency/inefficiency in their resource utilization for teaching and research separately, and teaching and research jointly. Inferences regarding the remaining 17 units should be made with some caution, since some of them turn from efficient into inefficient and vice versa, and hence are sensitive to the model specification. Furthermore, the results from the first group of models suggest that KTH units are more inefficient in teaching than research, meaning that, theoretically, they are able to educate more students given the same resources and research output. The comparison of the efficiency scores of corresponding models from three groups shows that a unit is more likely to be characterized as efficient by the joint model if it is efficient in either teaching or research or both, indicating that joint efficiency requires efficiency in at least one type of activity. Spearman s rank correlation coefficients table presented in Appendix I shows quite high correlation between technical efficiency scores of the respective models (above 0.88), suggesting that efficiency in teaching is likely to imply efficiency in research and also joint efficiency. This means that inefficiency in one type of activity is not offset by the other and hence teaching and research inefficiencies are complementary Technical and scale efficiency in the preferred joint model The previous section contains the results for three different groups of models differing in their output specifications. The models of the first group assume that university units are not able to alter their research output, and hence the research output is treated as non-discretionary, whereas the second group assumes that it is the teaching output that is non-discretionary. Both assumptions check for the robustness of results and provide insights into the operation of units. However, university units are in charge of both outputs and hence the complete model is the 22

23 one which includes both output categories as discretionary. Moreover, we believe that the best model should include not only quantitative but also qualitative indicators since university units are not homogenous in the quality of their outputs. The following analysis is based on the results of a model with 3 inputs and 6 outputs, which include both quantitative and qualitative teaching and research indicators, namely the number of full-year equivalent students in graduate and undergraduate education, the number of journal papers and conference papers, scientific quality and applied quality indicators. According to the results, shown in table 5, the mean performance of KTH units is relatively high. The average efficiency score is 0.96 with 74percent of units (35) operating on the frontier. Only 12 units are identified as being inefficient with an average efficiency score of 0.83, which suggests that on average they can turn into efficient via proportional increase of outputs by 17 percent. Table 5. Technical and scale efficiency of KTH units in research and teaching jointly (preferred model) Unit Number TE VRS TE CRS SE