Rankings as a Student Information Tool Presentation at the 20th Annual EAIE Conference, Antwerp September, 2008

Transcription

1 Rankings as a Stdent Information Tool Presentation at the 20th Annal EAIE Conference, Antwerp September, 2008 Leon Cremonini Centre for Higher Edcation Policy Stdies, University of Twente, The Netherlands l.cremonini@twente.nl Tel:

2 Contents of Presentation Why rankings? Approaches to rankings Indicators and critiqes Brief history of stdent information tools in the Netherlands Methodology of stdent information tools Pilot project cooperation Netherlands Belgim Germany Conclsion 2

3 For Whom is Ranking? For stdents: where shall I go and stdy? For HEIs: against whom am I competing? ; Who s the benchmark? For policy makers: Where shold special fnding be allocated? For society: where has my physician gradated? For employers: am I recriting the right person for the right job? Where did s/he gradate? 3

4 Existing Approaches to Rankings Who condcts the rankings? How are the rankings presented? How are variables pt together to create the ranking? 4

5 Who Condcts the Ranking? Magazines, e.g.: USNWR (US), Times (UK), Gardian (UK), Repbblica (I) Edcational pblishers, e.g.: Barron s (US), Good Gides (AUS) University-based Grops e.g.: Shanghai Jiao Tong University (China) 5

6 What Format of Rankings? Straight Nmerical Rankings Clsters or gropings, in which instittions are assembled into clsters of similar qality Top Level approach, Only the top instittions are ranked nmerically, while the worse performers are groped 6

7 How Indicators Are Presented Weight-and-sm approach Indicators are combined to prodce an overall instittional score. Each indicator is assigned a weight according to its perceived importance Netral comparisons (not weighted) All indicators mst be shown for all instittions Use of interactive technology 7

8 How to Rank and How Not An Example of a Widely Accepted Ranking clear, neqivocal positions clear rles for calclation of overall score 8

9 Can We Rank HEIs Like That? weights of indicators? rank scale with overall vale ranking of whole niversity? clear rank position is this niversity 64,8% as good as Harvard? 9

10 Most Used Indicators and Critiqes to Rankings (1) Reptation and Prestige: peer evalations throgh, e.g., srveys of other (similar) instittions presidents and deans. Sbjective measre Problem of circlarity New niversities are almost always ranked below the old Proximity (foreign niversities are less known) Except the sal top-10 Reptation and prestige privilege research-oriented 10

11 Most Used Indicators and Critiqes to Rankings (2) Stdent Characteristics, e.g. acceptance rates, high school class standing, entrance test scores, stdent diversity Reptational conception of qality, often adopted in HEIs admission policies Resorce notion of qality Problem: inpt measres fail to measre what the niversity provides to stdents 11

12 Most Used Indicators and Critiqes to Rankings (3) Staff characteristics, e.g. salary This variable mostly measres the instittional vale placed on research, for it is the best researchers (not teachers) who typically earn the highest salaries. Proportion of fll-time staff Proportion of staff with Ph.D. 12

13 Most Used Indicators and Critiqes to Rankings (4) Stdent faclty relationship, e.g. Stdent Staff Ratio and class size However, optimal class size is a fnction of discipline Stdent satisfaction Assets, e.g. fnding Withot information on how fnds are spent these measres are difficlt to interpret as measres of instittional qality 13

14 Most Used Indicators and Critiqes to Rankings (5) Expenditres How mch is an instittion committed to teaching/learning and/or research? Sitable for inter-instittional comparisons, bt what is actally inclded in the investments? Lack of standardised formats for reporting expenditres Can a HEI that spends more to achieve a similar reslt than another HEI be ranked higher for this very fact? 14

15 Most Used Indicators and Critiqes to Rankings (6) Stdent Grades/Test Scores Retention Rates and Gradation Rates Problem of grade inflation: rankings that rely heavily or entirely on this indicator favor instittions with lower academic standards over more demanding HEIs Gradate employment No control on inflencing factors e.g. social backgrond and connections 15

16 Most Used Indicators and Critiqes to Rankings (7) Research Bibliometrics (e.g. nmber of pblications and citations), Nmber of faclty with special awards (e.g. the Nobel Prize), Financial indicators of research (e.g. the research bdget, nmber of research grants etc.) and The nmber of patents issed. Moreover, some other rankings tilise also the nmber of Master and Ph.D. Other, sch as almni giving, etc. 16

17 The Dilemmas of Ranking (1) 17 Universities have different goals and missions Universities are internally differentiated: cannot be measred as a whole Bias towards research (low/no focs on teaching qality) Poplarity contests (appreciation vs. facts) Bias towards natral & medical sciences Bias towards English langage / Anglo-Saxon world Tables vary in their validity, comprehensiveness, comprehensibility, relevance, and fnctionality Rankings share broad principles and approaches, bt differ considerably in detail Variations related to: different aims, systems, cltres and availability and reliability of data

18 A Mltidimensional Ranking : The CHE Approach city, niversity stdents stdy otcome internationalisation teaching resorces research labor market, employability overall assessment (stdents, professors) 18

19 History of Stdent Information in the Netherlands 1980s: Minister spports data collection + book pblication Choice Gide Ca. 2000: web-based stdent information tools appear in Britain, Germany 2006: Netherlands web site appears: (English) : pilot in internationalisation: connect SK123 with German CHE- ranking and introdce CHE method in Belgim 19

20 Aims and Target Grops 1. Information for prospective / mobile stdents = primary target grop 2. Information for HEIs (benchmarking) Challenge: find a balance between redction of complexity for least informed stdents vs. detailed information for HEIs 20

21 Methodological Principles 21 Organised by disciplines / fields Mltidimensional Mlti-perspective Robst rank grops: Top Middle and Bottom grop No ranking of whole niversities No aggregated overall score Not jst one data sorce No sprios precision of leage tables

22 Major Data Sorces National statistics From state statistics agency From Ministry of Edcation Almni srvey Already collected Instittional data Paper Web form Stdent srvey Telephone web-based Collected on prpose 22

23 Data Sorces Not Used Reptation among professors E.g. Where wold yo send yor child? Note: only works well if Professors know their colleages Not internationally, except for stars Reptations relate to teaching performance In fact mostly research-based Accreditation stats It is a precondition for getting mentioned, therefore not a variable anymore 23

24 24 24

25 25 25

26 26 26

27 27 27

28 28 28

29 29 29

30 Pilot Project: NL BE - DE CHE approach is the better-known version of SK123 Eropean Commission wants to stimlate Eropean mobility of stdents Project aims: Link existing Stdent Info Tools (NL) Introdce CHE approach in a new contry (BE) 30

31 Experiences Pilot Project: Linking Stdent Info Tools Data collection differs in 2 contries Different administrative systems E.g. how to cont professors, stdents? Different political isses different ideas of what are relevant data/dimensions E.g. gender balance Different srvey approaches Ask for general opinions vs. ask for detailed data E.g.... (see next slides) 31

32 CHE Qestions on Corses 32 32

33 SK123 Qestions on Corses The extent to which the corse work material is interesting. (rate) The Level of the Stdy Material.(rate) 33

34 Experiences Pilot Project: Stdent Responses Low response rates Validity: is there a bias in responses? Comparability of jdgements Stdents have different expectation levels German stdents are satisfied with sitations that Dtch stdents find nsatisfactory Same also applies within a single contry Stdents in Amsterdam are more critical than stdents in Enschede This leads to bias of ranking Dtch stdy programmes are valed lower than German programmes In Netherlands: Enschede is the best town for stdents?? 34

35 Conclsion Stdent information tools are in high demand Stdents se web sites next to school advice and informal information Collecting data is hard work Validity is far from perfect International stdent information tools presppose demand for international mobility In Erope: small part of stdents, mostly at Master or Ph.D.-corse levels 35