Assessment of Research Performance in Biology: How Well Do Peer Review and Bibliometry Correlate?

Transcription

1 Assessment of Research Performance in Biology: How Well Do Peer Review and Bibliometry Correlate? BARRY G. LOVEGROVE AND STEVEN D. JOHNSON Bibliometric indices based on publishing output, and citation records used to measure scientific quality, are increasingly being employed to supplement and even replace traditional alternatives, such as the peer-review system. In this article we question whether peer review can predict bibliometric indices for individual researchers. We compared the ratings of scientific quality obtained using a peer-review system with the most popular bibliometric scores (h-, m-, and g-indices; total citations; and mean number of citations per publication) for 163 botanists and zoologists. Although the peer-review ratings were correlated with the bibliometric measures, they explained less than 40 percent of the variation in the scores. Most of this unexplained variation is presumably due to limitations of both the peer-review system and bibliometric scores. We propose a synergy between peer-review and bibliometric scores that can improve the assessment of scientific quality, especially by benchmarking peer-review decisions against bibliometric thresholds. Keywords: bibliometric scores, Hirsch index, peer review, citation record, publication record Scientists must regularly undergo evaluation of their performance for the purposes of promotion and grant awards, yet there is considerable controversy about the best means of obtaining information on the research performance of individual scientists. In the past, the peer-review system was considered a fair mechanism for measuring scientific quality (Kostoff 1997). Increasingly, though, this system is being supplemented, and even replaced, by bibliometric indices based on publishing output and citation records (Ball 2005, Hirsch 2005, Egghe 2006, Kelly and Jennions 2006, Lehmann et al. 2007). We pose several questions of interest to all researchers. First, how well does peer review predict bibliometric indices for individual researchers? Second, does peer review offer any obvious advantages over bibliometric scores, and vice versa? The latter question is critically important to grantawarding agencies, because bibliometric indices are obtainable almost instantly and, if screened appropriately for homonyms (different people with the same names) and synonyms (one person with different names and initials), should avoid human subjectivity. Individual scientists, too, want a system that provides the most accurate, and thus fair, evaluation of their performance that is possible. The National Research Foundation (NRF) in South Africa has long used a carefully managed system of international peer review to rate researchers according to their scientific performance. The rating evaluation is based purely on a scientist s past research performance, and does not involve assessment of project proposals or grant applications. Thus, we would expect a scientist s rating to correlate closely with his or her bibliometric score, because the rating is principally an assessment of the individual s research output and standing in the international scientific community. The NRF rating is the outcome of a two-stage process. First, applications are sent to members of subject-specific specialist committees composed mostly of invited NRF-rated peers and NRF management. For each application, the specialist committee chooses the names of at least six national or international reviewers. Half of the reviewers are selected from a list of potential reviewers supplied by the applicant, whereas the rest are selected independently by the specialist committee. Reviewers are requested to assess (a) the quality of the research-based outputs of the last seven years as well as the impact of the applicant s work in his/her field and how it has Barry G. Lovegrove ( lovegrove@ukzn.ac.za) and Steven D. Johnson ( johnsonsd@ukzn.ac.za) are with the School of Biological and Conservation Sciences at the University of KwaZulu-Natal in Scottsville, South Africa American Institute of Biological Sciences. 160 BioScience February 2008 / Vol. 58 No. 2

2 impacted...adjacent fields, and (b) the applicant s standing as a researcher in terms of both a South African and international perspective ( Between 2003 and 2006, for example, the NRF obtained 12,649 peer-review reports (59 percent from outside South Africa) to rate the research performance of 1554 researchers. These reviews are assessed by assessment panels of a specialist committee and NRF management, who agree on a rating. Scientists apply every five years to be rerated (see for additional information on the rating process). Although the NRF rating process has no bearing on a separate grant application process that typically follows a successful rating (nonrated biologists can also apply for NRF grants), the rating is the most important variable in a formula used by the NRF when allocating research funds to individual researchers. Each of the three main rating categories described below A, B, and C includes two to three subcategories (see for subcategory details). A yardstick for peer review All five bibliometric measures varied significantly among, and showed a significant positive relationship with, peerreview rating categories (figure 1, table 1). However, variance in these scores explained by ratings (R 2 ) was low, varying from 19.4 percent (m-index) to 38.5 percent (total citations; A rating: A-rated researchers are unequivocally recognized by their peers as leading international scholars in their field for the high quality and impact of their recent research outputs. B rating: B-rated researchers are those who enjoy considerable international recognition by their peers for the high quality of their recent research outputs. C rating: C-rated scientists are established researchers with a sustained recent record of productivity in the field who are recognized by their peers as having (a) produced a body of quality work, the core of which has coherence and attests to ongoing engagement with the field, and (b) demonstrated the ability to conceptualize problems and apply research methods to investigating them. Here we compare the NRF peer-review ratings of 163 botanists and zoologists with various bibliometric measures: the h-index the h number of papers with more than h citations each (Hirsch 2005); the m-index h divided by scientific age (years from first publication to last publication; Hirsch 2005); the g-index similar to h, but weighted by highly cited papers (Egghe 2006); the total number of citations; and the mean number of citations per paper, obtained from Thomson Scientific s ISI Web of Science ( We included the mean number of citations per publication in our analyses because this measure has been proposed to be better than the Hirsch index (Lehmann et al. 2007). Given concerns that citation practices might vary among scientific disciplines (Batista et al. 2006), we restricted our analyses to a sample of 163 biologists (botanists and zoologists) who were rated or rerated by the end of Figure 1. Frequency distributions of Hirsch s h-index for 8 subcategory ratings (C3 to A1) used by the National Research Foundation, South Africa, to measure scientific quality. The mean ± standard deviation of each subcategory is plotted at the top of each graph with the actual mean value. The subcategories within parentheses show the results of post hoc Tukey tests from the analysis of variance (table 1). These represent subcategories that are significantly different (p <0.05) from the value of the subcategory where they are listed. For example, the h-index of subcategory B2 is significantly different from C1, C2, and C3 (those in parentheses), but not from A1, A2, B1, and B3. February 2008 / Vol. 58 No. 2 BioScience 161

3 Table 1. Results of univariate analysis of variance of various bibliometric measures between National Research Foundation rating subcategories. Bibliometric F 7,162 R 2 p measure (ratings) h-index < m-index < g-index < Total citations < Mean citations < per publication Note: All data were log 10 -transformed to achieve homogeneity of variances. table 1). The percentages of h- and g-indices explained were identical (37.6 percent). Thus, peer review could at best explain 40 percent of the variation in the bibliometric measures of South African biologists. Post hoc Tukey tests showed that the mean h-index in a particular rating category was seldom significantly different from that in adjacent categories. The mean h-index of the B1 category, for example, was actually lower than that of the B2 category. Unlike the data from a recent report (Lehmann et al. 2007), our data suggest that the number of citations per publication is not a better estimator of scientific quality than the h- or g-indices. Put another way, our data suggest that scientists highly regarded by their peers are more consistently likely to have high h- or g-indices than high values for most alternative bibliometric measures, including the mean number of citations per paper. Hirsch (2005) argued that the h-index may provide a useful yardstick with which to compare, in an unbiased way, different individuals competing for the same resource when an important evaluation criterion is scientific achievement. The h-index is gaining popularity because it is considered one of the most robust measures of quantitative and qualitative research performance: it is insensitive to large numbers of poorly cited papers as well as small numbers of big-hit, highly cited papers. However, the latter supposed advantage may actually be a drawback of the index, because it does not recognize the value of papers with exceptional scientific impact (Egghe 2006). Egghe (2006) claimed that the g-index inherits all the good properties of the h index and...yields a better distinction and order of the scientists from the point of view of visibility. Like the h-index, the g-index is computed from a researcher s publication list ranked by decreasing number of citations; however, the g-index is weighted by exceptionally highly cited articles (Egghe 2006). The g-index requires more computational time than the h-index, but can still be computed by hand in about three minutes. In our data, there was a strong correlation between the g- and h-indices (R 2 = 0.905, p < 0.001; figure 2), but most of the unexplained 9.5 percent variance could be attributed to 3 of the 163 researchers, whose g-index exceeded the upper 95 percent prediction limit of the regression. The g-indices of Figure 2. The correlation between the h- and g-indices of 163 South African biologists. these researchers (rated A2, B2, and C2) were disproportionately higher than their h-indices because they were co - authors on a few papers with more than 200 citations. Given the similar variance between NRF subcategories explained by the h- and g-indices, the h-index and its variants should be the primary bibliometric yardstick, but calculating the g- index as well may help to identify outstanding individuals (sensu Hirsch 2005) in a discipline-specific context. Sources of peer-review and bibliometric variance Both peer-review and bibliometric scores have recognized sources of error and bias that can account for the large unexplained variance between NRF ratings and bibliometric scores. Although science is essentially founded on peer review, a recent study identified 25 different sources of bias in processes involving manuscript reviews and grant peer review (Daniel et al. 2007). Citing Shashok (2005), these authors emphasized that because reviewers are humans, their behaviour, whether performing salaried duties, enjoying their leisure time or writing reviews, is influenced by factors that cannot be predicted, controlled or standardized. Some recognized and contradictory sources of bias are gender, nationality, and scientific discipline (Daniel et al. 2007). Nevertheless, peer review of researchers is unlikely to be replaced entirely by bibliometric measures. The general correlation between bibliometric measures and peer-review research ratings found in this study and others (Bornmann and Daniel 2006, van Raan 2006, Daniel et al. 2007) is encouraging to advocates of either approach. Numerous refinements of the bibliometric indices have been published, and the h-index in particular is undergoing extensive scrutiny of its advantages and limitations (Batista et al. 2006, Bornmann and Daniel 2007, Jin et al. 2007). Some important limitations of the h-index are that (a) it does not control for scientific age, (b) it is sensitive to disciplinespecific citation patterns, and (c) if calculated from Thomson Scientific s ISI Web of Science, it is underestimated 162 BioScience February 2008 / Vol. 58 No. 2

4 because books and book chapters are excluded (Bornmann and Daniel 2007). Moreover, although it has been suggested that self-citations do not markedly alter the ranking patterns of researchers (Cronin and Meho 2007), the h-indexes for individuals with many cooperating coauthors can be inflated compared with those for researchers who work alone (Vinkler 2007). Nevertheless, these limitations can be overcome with additional research and computation of an individual s research output. For example, the implications of the lack of time cor rection in the h-index are that it cannot differentiate between active and inactive researchers, and that it always puts young researchers at a disadvantage relative to older, wellestablished counterparts. Indeed, the h- or g-indices never decrease; in a ratchet-like fashion, they either increase or remain the same with time (Hirsch 2005), and thus they cannot identify researchers whose productivity has waned. Hirsch proposed the m-index as a time-corrected h-index (Hirsch 2005), but the m-index is valid only if the h-index increases linearly with time, which is not always the case (Burrell 2007, Egghe 2007). Burrell (2007) advocated the calculation of the h-rate, the slope of the relationship between the h-index and scientific age (equivalent to the m-index when the slope is linear), as a meaningful indication of age-related research performance, whereas Jin and colleagues (2007) proposed the AR-index, which can increase or decrease with time in response to research outputs. Because many h-index-versus-time curves are not linear, the h-rate and AR-index can be calculated during slow and fast phases of a researcher s career. This approach could help explain, for example, why some B-rated South African biologists have h-indices higher than those of their A-rated counterparts: the B-rated researchers may have become unproductive later in their careers and subsequently become down-rated (information that can certainly be obtained through peer review, but also through close scrutiny of variations in the h-rate over time). For young scientists who have yet to establish clear biblio - metric patterns on the basis of citations, peer review will always remain an invaluable tool. Peer review is used, for example, by the NRF to rate young researchers (Y rating) and young researchers of exceptional potential (P rating). Similar systems that identify promising young scientists exist in other countries, and require an assessment process that does not depend heavily on citation profiles. A synergistic approach Despite the disadvantages of the h-index, its advantages cannot be ignored by research evaluators. First, it can be obtained very rapidly from several Web-based databases. Second, it is considerably cheaper to obtain, in terms of administration costs, although the Thomson Scientific ISI Web of Science is itself a very expensive service. Third, it avoids the perceived subjectivity, rivalry, nepotism, and secrecy of peer-review systems. Fourth, bibliometric measures are universally comparable within, though possibly not among, disciplines. Fifth, it provides transparency, allowing researchers to monitor progress in their careers and to identify what is needed to achieve personal targets. Clearly, there is merit in a synergistic approach whereby the rating of scientific quality through peer review can be benchmarked against bibliometric measures of performance within a discipline (Batista et al. 2006). It is difficult to accept that some of the glaring discrepancies evident in figure 1 can be attributed solely to the recognized limitations of the peerreview and bibliometric measures discussed above. These discrepancies include an A-rated biologist with an h of 15, lower than the mean h-index for the B1 and B2 subcategories, and a B2-rated researcher who has an h of 4, lower than the mean for all lower subcategories (B3 C3). On the other hand, the h-index of one unfortunate B2 researcher (h = 33) far exceeds the mean of both A subcategories. Hirsch (2005) advocated benchmark h-index ranges that can be used to separate successful scientists from outstanding scientists and unique individuals. In the disciplines of botany and zoology, the South African NRF rating categories of A, B, and C appear to correspond roughly to h-indices of more than 20, 10 20, and less than 10, respectively (figure 1). Indeed, we would argue that it is incumbent on the NRF, or any other body that evaluates researchers, to explain major deviations in rating patterns, and in this respect the h-index (and variants thereof) can prove a useful tool for identifying ratings that fall outside certain thresholds. These cases clearly deserve close scrutiny by peer evaluation panels and specialist committees. Acknowledgments We thank Lorne Wolfe and the National Research Foundation (NRF) in South Africa for comments on the manuscript. The NRF subcategory data were provided to the authors by the NRF under a contract of confidentiality. Three anonymous reviewers provided constructive suggestions that were implemented in the manuscript. References cited Ball P Index aims for fair ranking of scientists. Nature 436: 900. Batista PD, Campiteli MG, Kinouchi O, Martinez AS Is it possible to compare researchers with different scientific interests? Scientometrics 68: Bornmann L, Daniel H-D Selecting scientific excellence through committee peer review a citations analysis of publications previously published to approval or rejection of post-doctoral research fellowship applicants. Scientometrics 68: What do we know about the h-index? Journal of the American Society for Information Science and Technology 58: Burrell Q Hirsch index or Hirsch rate? Some thoughts arising from Liang s data. Scientometrics 73: Cronin B, Meho L Using the h-index to rank influential information scientists. Journal of the American Society for Information Science and Technology 57: Daniel H-D, Mittag S, Bornmann L The potential problems of peer evaluation in higher education and research. Pages in Cavalli A, ed. Quality Assessment for Higher Education in Europe. London: Portland Press. February 2008 / Vol. 58 No. 2 BioScience 163

5 Egghe L Theory and practise of the g-index. Scientometrics 69: Dynamic h-index: The Hirsch index in function of time. Journal of the American Society for Information Science and Technology 58: Hirsch JE An index to quantify an individual s research output. Proceedings of the National Academy of Sciences 102: Jin BH, Liang LM, Rousseau R, Egghe L The R- and AR-indices: Complementing the h-index. Chinese Science Bulletin 52: Kelly CD, Jennions MD The h index and career assessment by numbers. Trends in Ecology and Evolution 21: Kostoff RN The principles and practices of peer review. Science and Engineering Ethics 3: Lehmann S, Jackson AD, Lautrup BE Measures for measures. Nature 444: Shashok K Standardization versus diversity: How can we push peer review research forward? Medscape General Medicine 7: 11. van Raan AFL Comparison of the Hirsch-index with standard bibliometric indicators and with peer judgment for 147 chemistry research groups. Scientometrics 67: Vinkler P Eminence of scientists in the light of the h-index and other scientometric indicators. Journal of Information Science 33: doi: /b Include this information when citing this material. 164 BioScience February 2008 / Vol. 58 No. 2