Impacts of performance-based research funding systems: The case of the Norwegian Publication Indicator

Transcription

1 Research Evaluation 24 (2015) pp Advance Access published on 4 March 2015 doi: /reseval/rvv003 Impacts of performance-based research funding systems: The case of the Norwegian Publication Indicator Kaare Aagaard, Carter Bloch* and Jesper W. Schneider Department of Political Science and Government, Danish Centre for Studies in Research and Research Policy, Aarhus University, Bartholins Alle 7, DK-8000 Aarhus C, Denmark *Corresponding author. carter.bloch@ps.au.dk There has been a growing use of performance-based research funding systems (PRFS) as a policy tool. With the introduction of the Publication Indicator in 2004, Norway joined this international trend in which the allocation of basic funds is increasingly linked to performance indicators. The purpose of this article is to present and discuss the main results of a recent evaluation of the Norwegian Publication Indicator, which examines the Indicator s impact on publishing patterns, its properties, and how it has functioned in practice. This includes both a broad range of potential effects such as the Indicator s impact on the quantity and the quality of publications, Norwegian language publishing, and length of articles and monographs. It also includes an examination of properties such as the Indicator s legitimacy and transparency, how it functions as a measure of research performance across different fields, its use as a management tool, and how the system is organized and administrated in practice. In examining these questions, the article draws on a number of different data sources, including large-scale surveys of both researchers and research managers, multilevel case studies, and bibliometric analysis. The article concludes with a discussion of the implications of the analysis both for further development of the Norwegian Model and for PRFS in general. Keywords: performance-based research funding systems; bibliometric analysis; citation impact; research evaluation; funding. 1. Introduction There has been a growing use of performance-based research funding systems (PRFS) as a policy tool. Starting with the introduction of the Research Assessment Exercise in the UK in 1986, 14 countries had implemented a PRFS as of 2010 (Hicks 2012). Among the objectives behind this trend are the allocation of research funds to the most productive institutions, the pursuit of excellence in research, enhancing accountability of public research, and promoting greater alignment of research to societal and economic needs (Whitley and Gla ser 2007; Butler 2010; Hicks 2012). Funding systems are generally considered to be the most important instrument in steering research activities (Edquist 2003: 208). It is thus not surprising that funding in general and performance-based models in particular are among the most widely discussed topics concerning research policy. There are large differences in many of the systems in place across countries and, while some studies have been undertaken, our understanding of how these systems impact research is still very limited (Whitley and Gla ser 2007; Butler 2010). Generally speaking, current PRFS can be placed in one of three categories: peer-review-based models (UK), publication-based models (previously Australia, Norway, Denmark, and Finland), and citation-based models (Poland, Sweden, Slovakia, and Flanders in Belgium). Each of these models has its own strengths and weaknesses in terms of cost and organization, evaluation methods, transparency and legitimacy, extent of allocation of resources, and ß The Author Published by Oxford University Press. All rights reserved. For Permissions, please journals.permissions@oup.com

2 Impacts of performance-based research funding systems. 107 presumably also in terms of impacts on research and academic publication. With the introduction of the Norwegian Publication Indicator in 2004, Norway joined this international trend in which the allocation of basic funds is linked to performance indicators. The Norwegian Publication Indicator is a system for documenting Norwegian academic publishing with the aim of measuring publication activity and allocating research funding according to publishing performance (Schneider 2009). The Indicator distributes approximately 2% of total funding for the University and University College sector and thus plays a fairly marginal economic role. It is, however, well known that other factors than money play a role for the effects of PRFS. In particular, reputation has been mentioned as an important factor (Butler 2010). The purpose of this article is to present and discuss the main results of a recent evaluation of the Norwegian Publication Indicator (Aagaard et al. 2014). The evaluation of the Publication Indicator represents an interesting case for a number of reasons. First, the Indicator is a universal model with the same system for all disciplines, which at the time of its introduction was unique in an international perspective. Second, the Indicator has remained relatively unchanged since 2004, which greatly facilitates the evaluation of its effects. Third, an evaluation of the Norwegian case is interesting because a number of other countries have been greatly inspired by the Norwegian Model. Finally, the Indicator has been greatly debated in Norway with regard to its ability to achieve intended objectives and in particular with regard to potential unintended consequences. This article will, as with the evaluation itself, examine the Indicator s impact on publishing patterns, its properties, and how it has functioned in practice. This includes both a broad range of potential effects such as the Indicator s impact on the quantity and the quality of publications, Norwegian language publishing, and on the length of publications. It also includes an examination of properties such as the Indicator s legitimacy and transparency, how it functions as a measure of research performance across different fields, its use as a management tool, and how the system is organized and administrated in practice. In examining these questions, the article draws on a number of different data sources, including large-scale surveys of both researchers and research managers, multilevel case studies of selected institutions, and analysis of an extensive bibliometric database. This article thus adds to our understanding of the effects of PRFS. By providing important evidence-based nuances to our knowledge, it contributes to a literature which so far has been lacking these types of analyses. As Butler (2010) notes: Assessing the impact of PRFS is a fraught exercise, which perhaps explains the paucity of broad authoritative texts on the subject. The literature (...) contains relatively few concrete examples where the impact of PRFS has been examined in detail, either through investigative data analysis or well structured survey/qualitative investigations. The article is structured as follows. Section 2 introduces the Norwegian Publication Indicator, while Section 3 describes the evaluation in greater detail. Section 4 discusses the effects of the Indicator, while Sections 5 and 6 examine how the Indicator functions in practice and how it is used at different organizational levels. Section 7 concludes with a discussion of the main results of the analysis and their implications. 2. The Norwegian Model The Indicator was developed by the Norwegian Association of Higher Education Institutions (Universitets- og høgskolera det, UHR) in and is described in detail in the publication A Bibliometric Model for Performance-based Budgeting of Research Institutions (UHR 2004). It describes the purpose of the Indicator as follows: The Ministry s objective for this project has been to develop a qualitatively better, more reliable system of documenting academic publishing that will serve as the basis for the research component of the budgets for universities and university colleges. The purpose is to create a performance-based funding model for research and to encourage more research activity at universities and university colleges. The documentation system must also be flexible so that it can be used for other reporting needs and in individual and institutional publication lists, applications, project reports and evaluations, annual reports and information for the public at large (UHR 2004: 3). The indicator was first used to distribute funds to universities and colleges in 2006, and in 2008 the system was expanded with a unified database and classification system for the university and university college sector, health-care organizations, and the institute sector (Sivertsen 2008). Experience in Australia with an undifferentiated publication indicator where all publications count equally has shown that there is a significant risk for adverse publication behavior (Butler 2004). In the Australian case, their model led to a considerable shift in publication toward outlets with high acceptance rates and lower impact. In order to avoid this outcome and reduce incentives to increase publishing in low impact channels, the Norwegian Publication Indicator is differentiated, with publication channels classified in two levels. Level 1 comprises in principle all scholarly eligible publication channels, where eligibility criteria are some basic norms such as a standard external peer-review process. Level 2 is an exclusive number of publication channels which are deemed to be leading in a field and preferably with an international audience. Level 2 channels constitute at most 20% of a subject area s total scientific publications. The Indicator s

3 108. K. Aagaard et al. Table 1. The Norwegian Publication Indicator point system by level and category Category Level 1 Level 2 Journal articles 1 3 Monographs 5 8 Book chapters and articles in anthologies Source: UHR (2004). point system is weighted in terms of both level and publication form (journal articles, articles in anthologies, and monographs). For example, a level 1 journal article yields one point, while a level 2 article yields three points. Table 1 displays the Indicator s point system. Publication points for individual authors are fractioned according to the number of authors (e.g. for a level 1 journal article with four authors, each author contribution counts 0.25 points). Classification of publication channels is determined through a nomination process where both institutions and individual researchers may nominate channels. These proposals are then discussed in the appropriate committee for the subject area and approved (or rejected) by UHR s publishing committee. The Indicator itself is, however, only intended for use at an aggregated level. It is stressed in the background report that the Indicator is not an appropriate measure of research performance for individual researchers: A division of publishing channels in quality levels cannot replace or simulate qualitative assessments of publications at the individual level, just as a funding model cannot replace evaluations and strategies for research at the institutions. Publications of high quality are sometimes published in less reputable channels or vice versa. The channel cannot be used to draw conclusions about the individual publication or individual researchers, nor is this its purpose. (UHR 2004: 35). As indicated above, a number of system design choices are connected to the construction of any PRFS in the attempt to strike an appropriate balance between issues such as simplicity, transparency, and costs on the one hand and coverage, legitimacy, and validity on the other. This is by no means a simple task and involves a number of trade-offs. In the case of the Norwegian Model, the adoption of a general model enables comparison and redistribution of funds across all research areas. A clear strength of this approach is in particular the increased visibility of publication activity within the humanities and social science compared to funding models based solely on citation impact. The simplicity and transparency associated with this general model regarding to the allocation of funds 1 also helps to clarify the economic incentive structure, which is essential for the indicator s legitimacy and functioning. A general model, however, precludes the implementation of more customized models that are adapted to the diverse publication behavior of individual fields. A key question is accordingly whether it is possible to give scores to different types of publications in a fair and neutral manner across different disciplines, fields, and institutions? A related issue concerns the relation of the Indicator to research quality. As noted above, the Indicator is based on a two-tier ranking of publication channels, which are intended to reflect the status of the publication channels according to perceived quality criteria such as visibility, peer-review process, rejection rates, impact factors, etc. Hence, what is rewarded in the Norwegian Model is publication behavior and the underlying publication process, i.e. where you publish. However, in scientometrics, where you publish is usually disregarded in favor of the actual citation impact when it comes to measuring aspects of research quality (Gla ser and Laudel 2007). At the aggregate level, the publication channel will be correlated with citation impact, but this might not be the case for individual publications as the quote above also highlights. In the individual case, a publication published in a level 2 channel with little or no citation impact receives more publication points than a publication published in a level 1 channel which ends up having a high impact on the scientific community. The latter case is by no means rare. Even if the allocation of points on average is fairly accurate, this poses a threat to the legitimacy of the indicator in cases where the points of the overall model are used instrumentally at local levels. Paradoxically, the transparency, simplicity, and usability of the point system, which is seen as a clear strength of the Norwegian Model, are also the factors which could create a temptation for local managers and administrators to use it directly at disaggregated levels in spite of the recommendations quoted above not to do so. The limited economic importance of the Indicator, with only 2% of overall funding distributed through the Indicator, can be seen as a way of minimizing the scope of some of these issues. The assumption is that potential adverse effects on research behavior are minor and/or can relatively easily be remedied by other means. 3. About the evaluation The evaluation of the Norwegian Publication Indicator was conducted from July 2013 to January 2014 and was based on five sub-analyses:. Survey among researchers in the university and university college sector.. Survey among rectors, deans, and heads of departments in the university and university college sector.. Bibliometric analysis of the impact of the Publication Indicator and its properties.. Case/interview/document study of selected universities and university colleges.

4 Impacts of performance-based research funding systems Examination of the Norwegian Publication Indicator in an international context. The survey of researchers in the university and university college sector examines the effect of the Indicator on both the publishing patterns and research. This includes researchers understanding of the Indicator s effects, its characteristics, purpose, and use, as well as how the Indicator is used at individual and departmental levels. The target population includes all academic staff (including PhD students) currently employed at Norwegian universities and university colleges. The questionnaire was sent out to a sample of nearly 10,000 researchers, with a response rate of 34.4%. The survey among managers examines how the Indicator is used at different levels of the institutions. The population includes deans and department heads at universities and rectors and deans at university colleges. The sample consists of 219 managers, including 27 rectors, 31 deans, and 161 department heads. The response rate for the survey study among managers was 58.1%. Selected questions from the surveys of researchers and managers are included in the appendix. The bibliometric analysis examines the effects of the Indicator at the macro level. Publication analyses examine the extent to which the Publication Indicator has led to increases in publication activity, and eventual changes in publication patterns and general publishing behavior. The citation analysis examines the extent to which the Indicator may have affected the impact of Norwegian international journal publications. The bibliometric analysis is supplemented by economic data and survey data which together are used to examine whether the Norwegian Publication Indicator had the same negative impact on the research system as was the case with the Australian model in the early 1990s. Qualitative case studies consisting of a combination of interviews and desk research were also conducted for selected institutions at the institutional level, at the level of individual faculties, and at departmental levels. The purpose of these case studies is to provide a more detailed description of the various units internal use of the Indicator than the surveys alone can provide. The interviews and the desk research had a primary focus on how the Indicator influences the organization of research activities and how the Indicator was used at different levels in the institution. assessment of the effects of the Indicator and a series of bibliometric analyses. The bibliometric analyses are not designed as formal effect analyses, but instead examine developments in publication patterns during the period for which the Indicator has been in use. Central questions here are to what extent the Indicator has affected the volume of publications, whether there has been any change in publication impact, and finally whether there have been any changes in publication behavior over the period. The analysis draws both on the extensive publication database that was constructed as part of the Norwegian Publication Indicator and the Web of Science (WoS) citation database. The Norwegian publication database, which covers all fields and all forms of publication, is used to examine developments in volume and level of publications, while the WoS database is used to analyze citation activity among international journal articles within the natural, medical, and technical sciences. In particular, we examine developments in two indicators: the mean normalized citation score (MNCS) and the share of Norwegian articles among the 10% most highly cited within their fields (PPtop10%). A detailed description of these indicators and their construction can be found in Waltman et al. (2012). Figure 1 shows the development in publication points from the Indicator s implementation in to Total publication points for the UH sector have risen from 8,327 in 2004 to 15,189 in 2012, which amounts to a 82% increase over the period. This trend in publication volume is very similar when measured in number of (fractionalized) publications. As can also be seen in Fig. 1, the distribution of level 1 and level 2 points has remained very stable, with around 20% of total points 4. Effects on research and publication activity This section examines developments in publication activity and impact in the Norwegian University and University College (UH) sector and discusses to what extent they may be attributed to the Publication Indicator. In doing so, we draw on survey results concerning researchers own Figure 1. Development in publication output and publication points on the national level and distributed according to the two publication levels. Source: Norsk samfunnsvitenskapelig datatjeneste (NSD)/ Database for statistikk om høyere utdanning (BDH) [Norwegian Social Science Data Service\Database for statistics on higher education: action].

5 110. K. Aagaard et al. from publications in level 2 channels throughout the period. Hence, these aggregate developments do not indicate a shift in publication patterns toward channels with lower impact, as was found to be the case for Australia (Butler 2004; Sivertsen and Schneider 2012). Though, it should be noted here that the level of classification of publication channels has not been constant over the period and we thus cannot rule out that developments in the nomination process can have influenced these aggregate developments. Furthermore, aggregate developments within the fairly broad university and university college sector can include both changes in the number of researchers that actively publish and changes in average individual publication activity over the period. We return to both of these points below. At the institutional level, the share of publication points among the four main universities (University of Oslo, University of Bergen, University of Tromsø, and the Norwegian University of Science and Technology) fell from around 79% in 2004 to 68% in 2012, which in particular reflects strong increases in publication activity among more recently established universities and among university colleges, many of which have previously had a much weaker focus on research. This shift affects the allocation of funds as the pool of funding that is distributed by the Indicator has not increased over the period (aside from adjustment for price and wage increases). Hence, institutional allocations of funding are not determined by absolute numbers of publication points, but by the relative performance compared to other institutions. The fact that the funding pool has not increased in line with publication points thus means that an individual publication was worth 20% less in 2013 than in 2007 (the value of each point has decreased from 40,030 NOK in 2007 to 31,952 NOK in 2013). An additional aspect that needs to be taken into account when considering increases in publication volume is eventual changes in research resources devoted to the sector. Total research funding to the sector has also increased significantly in the period in question, though not nearly at the same rate as the publication activity. For example, R&D expenditures in the UH sector increased by 37% over the period from 2004 to 2011 (data for 2012 not available), while the number of R&D personnel increased by 21% from 2005 to Hence, even when taking developments in resources into account, the abovementioned increases in publications and publication points can still be considered to be substantial. As mentioned above, a key question here is whether increases in volume are due to a greater number of researchers that actively publish, to an increase in publication activity among those that were active throughout the period, or a combination of the two? Given that this evaluation was very broad-reaching, a rigorous detailed analysis of individual data was outside of its scope. However, some simple calculations have been conducted to help shed light on this question. Due to data limitations, we focus only on the four main Norwegian universities, where we have data on all researchers who have at least one publication record in a given year. The number of researchers with a publication in the four main universities has more than doubled from 2004 to 2012, with an increase of 116%. In comparison, the number of R&D personnel in these four universities increased by only 5% from 2003 to The share of researchers with at least one publication has thus increased greatly over the period. In terms of average publication activity (among researchers with at least one publication in a given year), the average number of publications increased by 26% from 2004 to 2012 while the average number of publication points fell by 19%. These few calculations are far from sufficient to fully describe developments in publication volume at the individual level. In particular, the average number of publications and points are based on an increasing number of researchers, which complicates their interpretation. However, it would appear very clear that the increase in publication activity reflects to a very large extent an increase in the share of researchers that actively publish in the academic channels covered by the Indicator. It is also interesting that the average number of publications has increased while the number of points has fallen. The result is likely due to increased cooperation, though shifts in types of publication can also have played a minor role. A central objective of the Norwegian Publication Indicator is to avoid a decline in impact for Norwegian research. To investigate this issue, we have examined developments in citation impact for Norway with results for Denmark, Sweden, and Finland included for comparison. Figure 2 shows the two abovementioned indicators of citation impact, the proportion of publications among the 10% most highly cited (PPtop10 %) and the MNCS. PPtop10% shows the proportion of a country s publications from a given year that is among the 10% most cited in the database. The indicator is normalized, meaning that Figure 2. Developments in impact, PPtop10%, and MNCS, for four Nordic countries from 2000 to Source: CWTS WoS database.

6 Impacts of performance-based research funding systems. 111 different publications, published at different times and in different subject areas can be compared. The statistical expectation would be that 10% of a country s articles for a given year will be among the 10% most cited in the database after a specified period (the number of years in which to count citations). In this case, we use a 4-year window, and for publications in 2009 and 2010, 3 and 2 years. The indicator s value represents the ratio between the observed number of publications among the most cited and the expected share of 10%. A value of 1 corresponds to the expected share, i.e. 10%, while a value of 1.2 means that the country has 20% more articles than expected among the most cited papers in the database from the current year. Distributions of citations of publications are highly skewed. A small share of publications typically receives the vast majority of citations. PPtop10% can therefore be interpreted as a measure of a country s ability to produce excellent research (Karlsson and Persson 2012). Note that the indicator is percentile-based and, contrary to average-based indicators, its calculation is not vulnerable to extreme values (articles with many or no citations). On the other hand, in the present case, we are indeed only interested in extreme values, as the PPtop10% focuses on the proportion of highly cited publications from the 90th percentile and up in the global citation distribution. The MNCS indicator shows the average citation activity for a country s articles. This indicator is also normalized. A MNCS value of 1 means that a country s articles published in a given year are cited to the same degree as the average for the database in the current year. A MNCS value of 1.2 means, therefore, that a country s articles are cited 20% more often than the average for the database. Note that the calculation of this averagebased indicator is much more vulnerable to extreme values in the citation distribution. It is clear from Fig. 2 that the two indicators follow each other for all countries, and for Norway and Sweden, the values for PPtop10% and MNCS are almost identical. For Norway, it shows that the articles published in 2010 have a PPtop10% of 1.07 and an MNCS of These are thus 7 8% above the database averages. From 2000 to 2004 we see a general increase, albeit from a relatively low level. From 2004 onward, there are fluctuations, but the indicator values are never below the level of 2002 and In other words, there is no indication in this time series that Norwegian citation impact in general has fallen since the Indicator s implementation. On the other hand, the values for both these indicators are lower than those in Sweden and substantially lower than those in Denmark. Furthermore, while the trend is at best weak, values have slightly increased over the period for the other Nordic countries, in contrast to Norway. Our overall approach in examining the impacts of the Indicator on research and publication activity was to utilize a combination of quantitative analysis of Figure 3. Survey results concerning researchers perceived impacts of the Publication Indicator (degree of agreement with statement). bibliometric data and survey data. This combination is particularly helpful in examining broader effects on publication behavior, many of which are difficult to examine solely on the basis of bibliometric data. In order to form the basis of our analysis of the effects of the Indicator on the publication behavior, we have studied researchers publishing patterns in general and the factors that play a role in their considerations for publishing. In the survey, researchers were asked to give their own assessment of how the Indicator has affected their research and publication activity. Figure 3 shows survey results of a series of statements about the potential effects. A relatively high proportion stated that they have a greater tendency to publish in level 2 channels. The same applies to writing articles rather than monographs, publications in international languages, and publications included in the indicator rather than other forms of communication. However, an examination of the Norwegian publication data available in the Cristin database indicates only very minor changes in the shares of different publication forms. For example, articles in journals or series accounted for 84% of the publications in 2005, 86% in 2008, and 84% in Shares for articles in anthologies and for monographs showed correspondingly little change over the period. Collaboration, both with national and international partners, is considered to be an essential element for research performance. There has been a concern that the fractioning of publication points in the Norwegian Publication Indicator would encourage less cooperation. However, we do not find any evidence of a negative impact on research collaboration from either the bibliometric analysis or the researchers survey. For example, when considering Norwegian articles in WoS, the share that was a result of collaboration between two or more institutions has increased from 58% in 2000 to 66% in 2004 and to 73% in A similar trend can be found for other Nordic countries such as Denmark, Sweden, and Finland. And, as Fig. 3 shows, a very small share of

7 112. K. Aagaard et al. researchers stated that the Indicator has led them to have fewer coauthors in their publications. An additional concern is that the Indicator s point system would encourage publication of research results in a greater number of shorter articles, a tendency that is often referred to as salami publishing. To examine this, we divided WoS journal articles into four categories according to their length: (1) 1 10 pages, (2) pages, (3) pages, and (4) over 30 pages. Not surprisingly, the vast majority of publishing activity is within the shortest articles in category A; however, the relative growth rates from 2004 to 2010 for the four categories are very similar. Hence, there does not appear to be a growing tendency toward publication in shorter articles. However, it should be acknowledged that the length of journal articles may be a somewhat poor indicator of this tendency, since salami publishing into multiple papers can be undertaken with no change in average length for the papers. An interesting question is to what extent these actual developments found in the bibliometric data correspond with data on perceived effects from the survey among researchers. The comparison is though difficult to make since, as noted above, the aggregate developments may be based on a number of trends at the individual level. Nonetheless, a number of the survey results correspond well with the bibliometric results, as is shown in Fig. 3. For example, a very small share of researchers (under 10%) stated that the Indicator had led them to collaborate less and only around 18% stated that the Indicator had led them to place greater weight on quantity than quality. Both these results fit well with the bibliometric data. Somewhat in contrast, 38% stated that they had a greater tendency to submit their work to level 2 channels, while the share of level 2 publications has remained constant throughout the entire period from 2004 to Researchers are, however, more negative concerning the risk of these effects on behavior in general terms. For example, 62% of researchers felt that the Indicator places too much weight on quantity compared to the quality of publications and 37% thought that the Indicator s point system reduces incentives for collaboration. It is important however to emphasize that very often we see a discrepancy between real and perceived effects. Quite often the perception, for example, elicited from survey responses, can be quite different from empirical evidence. For example, survey responses in relation to potential changes in publication behavior due the Research Assessment Exercises in the UK were clearly at odds with bibliometric results (Moed 2008). A similar tendency is the so-called Muhammad Ali effect which denotes the discrepancy between admitted own behavior and the expected behavior of others. The term originates from the psychological literature and is used to describe the finding that people see themselves as more moral than others (Allison et al. 1989). 5. Examining main features of the Norwegian Model This section reviews main characteristics of the Publication Indicator and how well they are perceived to function by researchers and managers. This includes the legitimacy and transparency of key features of the model such as the classification of levels, the weighting of different types of publications, and the nomination process for publication channels. An additional issue here is to what extent the Indicator is able to fulfill its intention of being universally applicable across different research areas. Both the functioning of the model itself and researchers and research managers perception of it can have an important influence on the Indicators impacts on publication behavior. With regard to the administrative organization of the Indicator, background research and interviews with key stakeholders highlighted issues such as transparency in the nomination process, the representation in individual committees, and the placement of channels among committees as important challenges. The greatest challenge appears to be found in relation to the nomination of channels at Level 2. In particular within research areas that are small or fall between several committees, it is often felt that there is a lack of influence on these nomination processes. It was also emphasized in interviews with stakeholders that the nomination processes can be characterized by conflicts of interest and elements of horse trading. The above findings are to a large extent supported by the survey results, which touched on both the Indicator s transparency and the nomination processes. For example, only 14% of the researchers view the nomination and designation process as being transparent. Managers are generally somewhat less negative than the researchers, as 32% view the classification process as transparent. Also, this share is rather low, however. Another important dimension in the discussion of the Publication Indicator s properties concerns the issue of neutrality across scientific fields. Full field neutrality in this context would mean that comparative researchers with the same ability, effort, and work conditions on average should earn the same number of publication points across disciplines. There are several potential factors of importance here. The first factor relates to the weighting of different publication types. For example, the humanities and social sciences typically have a larger tradition of publishing in monographs than the other disciplines. Similarly, the technical sciences typically have a greater emphasis on conference papers than the other areas. Another major factor is differences in the average number of authors per publication. Here, especially the medical and natural sciences typically have more authors per publication than is the norm in the humanities and much of the social sciences. Finally, as a third key factor, there may be potential differences across fields in the average time required to produce a publication, and

8 Impacts of performance-based research funding systems. 113 there may also be differences in how difficult it is to get publications accepted in Level 2 channels. This last point is beyond the scope of this analysis, but can potentially play a role. To investigate field neutrality, we used the bibliometric data to compare publication points for researchers who can be considered comparable across disciplines. We chose to compare researchers in the same academic positions at the Universities of Oslo, Bergen, Tromsø, and NTNU. The analysis was limited to the four major universities in an attempt to ensure that working conditions across institutions are comparable. For all academic levels, we found substantial differences in both mean and median values for publication points across disciplines. For example, the average is 2.5 times higher for professors in humanities than for professors within medicine and health and about 3 times larger for associate professors and postdocs. These results are consistent with those of Piro, Aksnes, and Rørstad (2013). Taking this at face value and interpreting publication points as a universal measure of performance across fields, the results imply that researchers in the humanities are 2 3 times more productive than researchers within medical and natural sciences, 5 which does not seem reasonable. These large differences in average points across fields thus strongly indicate that the Indicator has not met the aim of ensuring full field neutrality. 6. Implementation and internal use of the Indicator at institutional level As mentioned above, the Indicator is a system to allocate research funds at the institutional level. As the model only is designed for use at aggregated levels, all institutions are accordingly encouraged to develop more appropriate local models that take the overall objectives of individual departments into account. Nonetheless, it is also clear that the Indicator is an incentive model that ultimately must influence the individual level if it is to have an effect. This is a central dilemma for institutions. There will, on the one hand, be problems associated with a direct use of the model locally, because the Indicator is too broad and imprecise to adequately measure publication performance on an individual level. On the other hand, however, the introduction of local models will require resources to design and implement, and can potentially lead to a decline in income linked to publishing points in cases where local incentives differ from the general model. In any case, the implementation of local models can have important implications for both effects and legitimacy. This section examines the use of the Indicator at the institutional level, at the level of individual faculties, and at departmental and individual levels, relying both on case study interviews and surveys of researchers and management. The analysis not only looks at the economic redistributive consequences of the indicator but also at how it influences career and wage decisions as well as how it is used for monitoring and goal setting. Obviously, an important aspect concerns the direct economic incentives of the indicator at different levels of the institutions: how and to what extent is funding allocated down to the faculties 6 and further down to the departmental levels? The survey results as well as the case studies show that there is a very wide variation in practices both between individual institutions and internally within these institutions. The overall picture is, however, that the Indicator s direct economic incentives in most cases are weakened within the institutions so that the value of each publishing point is significantly lower for an individual department than for the institutions as a whole. Though, there are exceptions to this. For example, one faculty examined in the interview study chose to increase the economic incentive of the indicator by a factor of four for individual departments in an attempt to improve performance. However, in spite of this general weakening of the direct economic incentives of the Indicator at departmental levels, the interviews at the management level and the surveys among researchers and department heads showed that the Indicator is used in a number of other important ways, both at the departmental level and in measuring performance of individual researchers. An important part of this is the way in which the Indicator is used in relation to wages, bonuses, and promotions. Figure 4 shows the results of the surveys of researchers and department heads. Slightly more than 20% of the department heads indicate that individual wage allowances are awarded on the basis of publishing points and slightly less indicate that publication points are used as an explicit criterion to qualify for salary increases. In more general terms, more than 40% state that the indicator is used as one of the assessment criteria when negotiating salary and bonuses. A similar pattern applies to the researchers, though compared to the heads of departments the Indicator is generally given slightly less importance for these issues. Figure 4. Use of the Publication Indicator at individual and departmental levels. Survey responses from researchers and department heads (% that either strongly agree or agree with statement).

9 114. K. Aagaard et al. Another important link between the central model and the actual use at the local level is the way in which it is utilized with regard to monitoring and as a means of setting strategic goals. Both the survey results and the case studies show that the Indicator appears to play a rather important role in these respects. The survey shows that almost 90% of all managers state that the indicator is either somewhat or greatly used for surveillance/monitoring of research activities. If we consider the same question among the researchers, we see that the shares here are slightly lower, but still quite high. About 66% of the department heads state that they use the Publication Indicator to assess individual researchers publishing performance. Similarly, 73% use the Indicator for internal analysis of publication activity. There are also clear trends concerning the extent to which the Indicator is used to form the strategic objectives for publication activity. Around 86% of all managers consider the Indicator to be either very or somewhat important to the units strategies and objectives. An equally high share of department heads, 87%, state that they have targets for increasing the number of publication points, and 51% go even further and set specific publication goals measured in average points per researcher. The overall picture is accordingly that the direct economic consequences of the Indicator at local levels are marginal, but that the incentives nevertheless appear to trickle down to the individual level through a number of different mechanisms. What we observe is also a very high degree of variation across institutions, fields, and subunits with regard to the local implementation, and a limited attendance to the potential problems associated with a direct use of the indicator at disaggregated levels. As one of the respondents in the interview-study, a Vicerector of one of the larger universities, formulated it: no institutions have done a good job in creating appropriate local models (our translation). It is in other words acknowledged by several managers that this task so far has been somewhat neglected at most institutions, and that limited coordination and communication takes place between levels in terms of discussions of how the Indicator can be used best. 7. Discussion and conclusion When the Norwegian Indicator was introduced in 2004, it represented a unique and innovative contribution to the international development of PRFS. The Indicator situated itself between the cost-intensive peer-reviewbased models, the citation-based models with coverage problems, and the easily manipulated, undifferentiated publication-based models. The Norwegian Indicator in particular stood out in this context with its universality, with its full coverage across all disciplines, with its differentiated allocation of points, and with its high degree of transparency and relatively low costs. However, neither the Norwegian Indicator nor other known PRFS can be characterized as optimal models. All system designs represent different trade-offs. This article has, through analysis of the Indicator itself, how it has functioned as a measure of research performance and actual developments in publication activity, highlighted a number of the trade-offs associated with the Norwegian Indicator. This conclusion discusses the main results of the article. The bibliometric analyses find strong increases in publication output from 2004 to the present in terms of total number of publications, publication points, and journal articles in the WoS. These findings are in accordance with general findings from UK, Australia, and Spain, where implementation of PRFS, irrespective of the model used, seems to have triggered an increase in publication output (e.g. Jimenez-Contreras, Anegon and Lopez- Cozar 2003; Butler 2004, 2010; Moed 2008). Nevertheless, causality and marginal effects are difficult to establish. Good counterfactuals are needed, such as research sectors not included in a model (Butler 2004). Yet, for the present Norwegian case, we do not have such obvious counterfactuals. No doubt, the rise in output is a complicated function of several factors where especially more resources going into the Norwegian science system during the period examined seems an important one. But lack of sufficient data prevents us from examining these different factors in detail. Nevertheless, the macro data on academic manpower we actually do apply, in our view, suggest that the resources going into the Norwegian system cannot alone explain the substantial rise in output. While the increase in new researchers certainly contributes to the rise, so does the increased activity of researchers already in the system; some begin to publish and some publish more actively. Whereas publication output has increased considerably, the overall development in citation impact for Norway in the period from 2004 to 2010 remains basically stable just above the database average. In other words, there is no indication in these macro data time series that Norwegian citation impact in general has fallen or risen since the Publication Indicator s implementation. These findings are different from the experiences with a similar model in Australia. It is well-documented that in the Australian case, national citation impact dropped as a direct consequence of the models incentives to focus solely on productivity (Butler 2004). Such a development was a concern in Norway and a major reason for construction of a two-tiered classification of publication channels. A drop in impact has so far been avoided, but on the other hand, there is also no discernible increase in impact. Under these relatively stable macro results we do, however, find indications of other potential problems which threaten the legitimacy of the Indicator. These issues are in particular related to the aim of ensuring full

10 Impacts of performance-based research funding systems. 115 field neutrality, the aim of creating transparent and legitimate nomination processes, and the aim of replacing the overall Indicator with more appropriate models at local levels. As shown, the analysis documented a significant variation in the average points per researcher across disciplines indicating that the intention to create an Indicator that is neutral across different fields has not been achieved. A central question in this context is whether the problem of lack of neutrality has a scope that requires a solution. In terms of economic scale it can be argued that only a relatively modest amount of funding is transferred between fields due to a lack of neutrality. From this point of view, it can be considered to simply accept the imbalance. However, while the economic impact is probably small, the problem may still have significance for the Indicator s overall legitimacy, which merits further attention. Also with regard to the nomination processes, challenges were identified in relation to transparency, representation in individual committees, and the placement of channels among committees. Our material does not allow us to draw clear conclusions in relation to the problem s precise nature and scope. Regardless, also there appears to be a need to address the lack of legitimacy that can be observed. Finally, the analysis indicate that the abovementioned problems often are accentuated at the institutional level due to limited efforts to create more appropriate local models. More generally, the analysis thus finds that it appears to be quite difficult to maintain that the Publication Indicator is only designated for use at an aggregated level. As a result, there appears to be considerable uncertainty as to how to use the Indicator appropriately at the lowest levels of the institutions. One explanation for this is that the challenges and costs associated with designing local models are ultimately borne by each unit. Each department is, so to speak, alone with the problem of how to deal with the Indicator, and there seems to be very limited coordination and communication between levels in terms of how the Indicator can be used best. Thus, we see potential benefits from a more active and open sharing of experiences on both vertical and horizontal dimensions in the system. Acknowledgments This article draws on analysis from the Evaluation of the Norwegian Publication Indicator and has thereby benefited from a number of people that assisted with or provided input to the evaluation, including Gunnar Sivertsen, Vidar Røeggen, Dorte Henriksen, Per S. Lauridsen, Thomas K. Ryan and the evaluation steering group. This article has also benefited greatly from comments by two anonymous reviewers. Funding Financial support from Aarhus University s Senior Management Strategic Funds for the project Contextualizing and Measuring Research Performance is gratefully acknowledged. Notes 1. Each point earns a specified amount that is the same for each institution. The price of a publication point is determined each year based on the total number of publication points produced for the sector as a whole is included as this is the first year of publication counting, though budget allocations did not in fact begin until Source: NIFU: nifu/?language=no (accessed April 6, 2014). 4. Source: NIFU: nifu/?language=no (accessed April 6, 2014). Statistics are not available for Notice, many researchers in Norway from the medical fields have double affiliations, a university and a hospital. This is reflected in the fractioning of the publications and therefore also in the calculations relating to the degree of field neutrality. 6. Both here and below, by faculty we mean an administrative grouping of departments and centers (e.g. Faculty of Arts and Sciences). References Aagaard, K. et al. (2014) Evaluation of the Norwegian Publication Indicator (in Danish). Oslo: Norwegian Association of Higher Education Institutions. Allison, S. T., Messick, D. M. and Goethals, G. R. (1989) On Being Better but not Smarter than Others: The Muhammad Ali Effect, Social Cognition, 7/3: Butler, L. (2004) What Happens When Funding is Linked to Publication Counts?. In: Moed, H. F., Gla nzel, W. and Schmoch, U. (eds) Handbook of Quantitative Science and Technology, pp Dordrecht: Kluwer Academic Publishers.. (2010) Impacts of Performance-based Research Funding Systems: A Review of the Concerns and the Evidence, Performance-based Funding for Public Research in Tertiary Education Institutions: Workshop Proceedings, pp Paris: OECD. Edquist, O. (2003) Layered Science and Science Policies, Minerva, 41: Gla ser, J. and Laudel, G. (2007) The Social Construction of Bibliometric Evaluations. In: Whitley, R. and Gla ser, J. (eds) The Changing Governance of the Sciences, Vol. 26, pp Netherlands: Springer. Hicks, D. (2012) Performance-based University Research Funding Systems, Research Policy, 41/2: Jimenez-Contreras, E., Anegon, F. and Lopez-Cozar, E. (2003) The Evolution of Research Activity in Spain -The Impact of the National Commission for the Evaluation of Research Activity (CNEAI), Research Policy, 32/1: Karlsson, S. and Persson, O. (2012) The Swedish Production of Highly Cited Papers. Vetenkapsradet. <Stockholm. se/documents/vetenskap_samhallet/forskningspolitik/2012/ akademirapport_breakthrough_research_ pdf>.