Reference classes: A ranking-based benchmarking tool. Nicolas Carayol* ; Ghislaine Filliatreau, Agenor Lahatte

Transcription

1 Reference classes: A ranking-based benchmarking tool Nicolas Carayol* ; Ghislaine Filliatreau, Agenor Lahatte Observatoire des Sciences et Techniques (OST) *GREThA, Universite Bordeaux IV - CNRS Shanghai- WCU3- nov09 OST was founded in 1990 dedicated to the design and the production of quantitative indicators on R&D What have we learnt? 1) Tools ARE important to support /allow policies 2) Diversity of tools is required to sustain diversity of institutions and to help them to differentiate 1

2 Here we propose a new methodology to compare, rank and benchmark universities The basic principle We order pairs of universities by a «dominance relation» the computation used to establish this dominance relation is customizable : it gives you the possibility to choose the ranking principles according to which universities will be compared the ranking principles are designed to reflect some of the «values» that a policymaker would want to give to different aspects of the outcomes of the universities As a calibration exercise, we apply this methodology to French Universities, using the impact of their scientific publications Each «customized ordering exercise» will produce : 1) a relative positioning tool : «ordered pairs» of universities pairs of universities are ordered by dominance ; provides, for each university, on one hand the universities that dominate it, and on the other hand the universities that are dominated by it 2) a benchmarking tool : «classes of reference» for each university, its «reference class» will include its «peers» that is thoses universities that don t dominate neither are dominated by it 3) a global ranking tool : «dominance pseudo-ranking» : the rank of each university will be given by the number of universities that it dominates 2

3 The underlying theory comes from economists, to be used to compare the outcomes of a set of agents : 1) the outcome is made of unitary elements (e.g. publications), 2) each element could be described by a «quality» index (e.g. the impact of the publications), 3) the policymaker is concerned by both quantity and quality, 4) the policymaker does not know exactly the function which transforms the quality and the quantity in valued outcome, but he is able to refers to general principles for this valuation. First choice Three different proxys used for measuring the impact : either the number of citations obtained by the publication Or the JIF (the impact factor of the journal of the publication) Or the normalized JIF (in order to give more weight to the small, not higly-cited scientific specialities) 3

4 Second choice Three types of dominance relation : Strong dominance Dominance (positive value of impact) Weak Dominance (increasing value of impact) The more the better Impact B A N 4

5 A B Select a part of the distribution Making an assumption on the valuation function : e.g.you decide that you are interested only in those publications which are cited 5

6 Weak dominance : additional assumption on the valuation function e.g. you want it not only increasing, but also weakly convex («I give more price to a publication with 10 citations than to 2 publications with 5 citations) Third choice Choose the level of selectivity Another parameter is Φ, used to select the part of the scientific production you want to take care of : for instance, if Φ=0.1, then you will consider only the publications which belong the 10% most cited publications in the world 6

7 Fourth choice Choose to consider disciplinary (department) or interdisciplinary (whole university) ranking The data used for calibration French HEIs (129 institutions) The institutions checked the validity of the signing patterns. OST in-house ISI-WOS database recording of publications and citations Humanities and social sciences are excluded. Large disciplines : 1/ Fund. bio., 2/ Medicine, 3/ Ap. bio/ecol., 4/ Chem., 5/ Physics, 6/ Science univ., 7/ Eng. sciences, 8/ Maths. Multidisciplinary sciences papers published in PNAS, Science & Nature have been allocated to their reference discipline Three-years publications ( ) and 3-years citations window. 7

8 Graphic interface and easy-to-grasp visualizations One example for 10 top-universities : dominance network reference classes and pseudo-ranking 8

9 9

10 Reference classes : sets of peers of.changes with the proxy used 10

11 Highly selective Changes with the type of dominance relations used Conclusion From a theoretical model, we try to design a flexible tool to make the policymakers able to «play with» the values they would give to various aspects of the university outcomes The exercise has been calibrated with French universities, to test the relative positioning of an university among the others, to identify its «peers» in order to benchmark with and to rank it globally The next question will be about the pay off of using this tool Next steps : check the effect real use by university staffs, extend the exercise to more European HEIs, use other data (graduates, contract made with private companies) 11

12 Carayol, N., Lahatte A., 2009a, Dominance relations and universities ranking, mimeo. Carayol, N., Lahatte A., 2009b, The academic value of knowledge : The ranking of various actors and research institutions, mimeo. Carayol, N., Filliatreau, G., Lahatte A., 2009, Reference classes : A ranking based tool for benchmarking universities research. Thank you for your attention 12

13 13