COMMISSION OF THE EUROPEAN COMMUNITIES Brussels, 12.3.2001 SEC(2001) 434 COMMISSION STAFF WORKING PAPER How to map excellence in research and technological development in Europe
TABLE OF CONTENTS 1. INTRODUCTION... 3 2. OBJECTIVES AND EXPECTED RESULTS... 3 3. IDENTIFICATION AND MAPPING OF EXCELLENCE... 5 3.1. Possible approaches... 5 3.2. Constraints... 6 4. IDENTIFICATION OF S&T FIELDS... 6 5. PROCESS AND METHODOLOGICAL ELEMENTS... 7 6. ROLES OF THE HIGH LEVEL GROUP AND THE COMMISSION... 9 6.1. Role of the HLG... 9 6.2. Role of the Commission services... 10 7. TIMETABLE AND FUTURE DEVELOPMENTS... 10 2
1. INTRODUCTION In January 2000, the Commission adopted a Communication proposing the creation of a European Research Area 1. This project offers a new horizon for scientific and technological activity and for research policy. It essentially aims at creating favourable conditions to increase the impact of research efforts by strengthening the coherence of research activities and policies conducted in Europe. In particular, it recognises that world-class excellence exists in practically all areas and disciplines in Europe. These competencies, however, are not always sufficiently well known across national borders, for example by companies. At the Lisbon European Council on 23-24 March, the Heads of State or Government endorsed this project and set a series of objectives and an implementation timetable. In particular, they requested to map by 2001 research and development excellence in all Member States, in order to foster the dissemination of excellence. Subsequently, the Research Council resolution adopted on 15 June called on the Commission, in close co-operation with the Member States and taking into account national research plans and policies, to present to the Council, before the end of 2000, objectives and a methodology, including indicators and peer review, with a view to mapping excellence in all Member States by the end of 2001. In order to ensure close co-operation with Member States on the subject of excellence, Commissioner Busquin convened a group of nominated representatives from Member States, the High-Level Group. Following detailed preparatory work by the Commission services, discussions were held with the High-Level Group on 20.10.2000, 24.11.2000, and 13.12.2000. On 23 November, a specialised workshop was organised, where experience in various countries was presented regarding methods for assessing excellence. The present document takes into account the lessons from such experiences, as well as the views expressed by the HLG members. 2. OBJECTIVES AND EXPECTED RESULTS Mapping excellence in research and technological development (RTD) at a European level aims at providing useful intelligence allowing greater advantage to be taken of the potential of excellent competencies available across the EU: (1) Based on the identification of specific RTD capabilities existing in Member States, including less known and/or small ones, assessment of their excellence should make it possible to identify excellent competencies. This should allow visibility raising across borders, by disseminating the mapping results widely through public web sites (e.g. Europa, Cordis), through EU and national innovation support agencies, etc. A wide range of users would be targeted: industry (with special attention for SMEs) and investors, including from third countries; policy makers; RTD managers; public services, etc. Ultimate effects could show as intensified networking, increased intra-european mobility and knowledge transfer, and greater attractiveness of Europe. 1 COM (2000) 6 final 3
(2) Further S&T field-oriented analysis and comparison with global developments could provide evidence on evolving patterns of excellence in Europe, including sectoral strengths and weaknesses. This information should provide a basis for generating and structuring the critical mass of competence needed to meet strategic European challenges. Relevant follow-up actions by the MS and the EU might concern targeted "open co-ordination" between national policies, and specific support for the networking of and around identified excellent competencies. (3) Further analysis of the excellent competencies identified and the environments in which they operate could provide evidence and good practice for disseminating excellence throughout the regional, national and European innovation systems. Such information would introduce a learning process allowing public authorities, where opportune, to adapt funding schemes and regulatory environments in order to foster excellence in a general way. Whereas objective 1 can be addressed already in the short term, objectives 2 and 3 clearly refer to mid or long term action. Proposals for follow-up actions should be elaborated in the course of 2001, so that the first "excellence maps" are exploited optimally. In this context, it is important to note that the above objectives are not directly related to funding from any side, but that the information provided as result of the exercise may be used by all players in the innovation system as they see fit. The main impact is expected from action by governments and industry in MS. Concretely, the result of the mapping exercise should be the identification of an adequate number of entities in each of the selected S&T fields (see 4 below). While the exercise is not meant to identify individuals, mapping will only achieve full value by identifying organised groups of researchers such as institutes, companies, departments, networks, even when they are small. Furthermore, such entities could be located in all possible settings: universities, public or private research establishments. RTD entities in industry should be included unless their competence is considered commercially confidential; entities having excellent competence should be identified not only in classical S&T fields, but also, as far as possible, in emerging disciplines and interdisciplinary fields; identification would be based on the entity's performance in research and technological development 2. This would principally reflect human competence. The issue of infrastructure of European interest is being addressed in parallel through another action in the framework of the European Research Area; qualification for excellence would be based on the quality, rather than the quantity level of the entity's performance in RTD. For this purpose, the mapping methodology could set thresholds by S&T field. In any case, qualification for excellence would be determined with reference to the world level, in order to assure EU relevance and wide acceptance of the outcomes; 2 RTD entities may be excellent in many dimensions, e.g. in research, training, impact on innovation in Europe, or organisation. Studies show a good degree of correlation between excellence in different dimensions. However, for the objectives of this exercise, it seems best to avoid any confusion with "benchmarking research institutes", by focusing on performance in RTD. As a consequence, the entities presented in the resulting maps will not necessarily have the excellence in other dimensions that would permit the label "centre of excellence" in its usual connotation. 4
presentation of the results should aim at offering a maximum of practical information, so as to facilitate "open co-ordination" and networking initiatives. Dissemination activities, to be further elaborated, need to be adapted to the different types of potential users, such as industry, academia, policy makers, public services and other interested parties. Results would for example be displayed on the Internet as user-friendly "maps" providing some information on the entities concerned and the nature of their competencies, as well as direct links to the entities' own web sites. To gain the necessary broad acceptance, the methodology has to be: reliable and robust in very different S&T fields and across European countries; objective by relying as much as possible on quantitative data and independent expertise. This should ensure that the outcomes are comparable across very different national S&T systems; transparent regarding inputs, methods, and outcomes. Moreover, the methodology should: be applied repeatedly, to provide a dynamic picture of excellent competence as it evolves over time. Mapping should become a process producing useful results from the start, with regular updates to ensure continued validity; build on relevant national experiences with assessing excellence. The scope for learning and improving the methodology itself in later exercises should be fully exploited; involve the stakeholders in European RTD. Early information on the process should be spread widely especially towards the scientific and industrial communities. Input or feedback should be solicited as appropriate not only from the HLG, but also from relevant representative organisations established at European level. 3. IDENTIFICATION AND MAPPING 3 OF EXCELLENCE 3.1. Possible approaches The approaches outlined below to identify scientific and technological excellence have been experienced in some EU countries and used in evaluation studies in particular. They might be applicable for an EU-level exercise in certain regards. One can think about a complete bottom-up approach, widely asking the scientific and industrial communities to submit a set of data and information on a voluntary basis and assess it appropriately. This methodology could deliver reliable results and involve the scientific and industrial communities from the beginning. However, the magnitude of the task appears very substantial at EU level considering the resources and time available. The workload for these communities themselves may also be too high, especially considering that no financial reward is foreseen, thus endangering the success of such a method. Implementation would require a share of tasks between EU and Member States at both financial and working levels, as well as a tight co-ordination. 3 The term 'mapping' is predominantly used as a technical term in bibliometric analysis for visualising S&T structures. In this exercise however, mapping reflects the identification of S&T excellence and its geographic display. 5
Another approach could be based on a survey, where representative panels of the different scientific and industrial communities and possibly other relevant organisations would be requested to name, in their field of competence, say, the top 10 research entities in their country and the top 10 abroad. This approach could be feasible considering the time and resources available. However there are two significant disadvantages. First, although it could provide valuable information by comparing and analysing the preference and knowledge of the different panels, it is based entirely on subjectivity. Second, it might reflect a deep national/regional bias that might considerably reduce the credibility at a European scale. Another possibility could be based primarily on the combination of two kinds of indicators: input indicators like the number of scientists employed and output indicators like the number of publications. Although very different, both sorts of indicators are valuable to appreciate excellence. However, the links between the different factors composing excellence and the question of why some entities are better than others are difficult to capture through a sole indicator s based approach. In order to gain scientifically sound insights, as well as to respect the tight time frame imposed, an appropriate methodology would build on several elements taken from the above mentioned methodologies. This approach could be called a modulated indicators-based approach. It represents the Commission services preferred approach and as such, is developed in more detail below. The different approaches presented above have been discussed with external expertise, the internal Commission working group on excellence and the other Commission services concerned, as well as with the HLG. 3.2. Constraints The actual mapping work cannot be performed by the Commission services themselves. Therefore, calls for tenders will have to be launched as soon as possible. If launched at the beginning of 2001, this process will take until the Summer 2001 to be completed, reducing the actual work span for contractors and the necessary reviews to five or six working months only, if the maps have to be finalised by the end of 2001. In order to be of value to policy makers, the method has to be feasible and its results have to be acceptable for the scientific and industrial communities involved. In view of the complexity of the task, mapping all possible fields until the end of 2001 would run the risk that the results would be below the expected quality and could therefore jeopardise the overall credibility of the exercise, leading to possible adverse reactions from the stakeholders. It is therefore proposed at this first stage to limit the S&T fields to be mapped until the end of 2001. 4. IDENTIFICATION OF S&T FIELDS The choice of fields to be mapped is a crucial question. It would have been convenient to base this choice on a recent European-wide foresight exercise. Unfortunately, such a project has not been undertaken yet. A consensus among the HLG members resulted in the limitation of the areas to be mapped to Life sciences, Nanotechnologies, and Economics. Such a pilot approach will produce only a limited number of maps in a first stage but will enable the parties involved to learn about methodological advantages and disadvantages and incorporate the learning effects into a refined methodology, which can then be used to continue the mapping from 2002 on. At the same time, the pilot exercise should lead to usable and interesting results. For this purpose, a single map on a broad 6
area like Life sciences would not be very meaningful. A screening based on disciplines and assigned journals will be used in order to identify a number of hot, possibly non-classical, interdisciplinary research fields, which are characterised by rapidly increasing output rates and a significant scientific impact. The EU policy relevance of the fields will also be considered. For the necessary selection of the fields to be analysed in 2001, the HLG members will be consulted to take fully account of existing evaluation schemes or priorities in their respective country. Considering the constraints mentioned in 3.2, the exercise should be limited to approximately ten fields spread over the three areas. It is fully acknowledged that this pilot exercise is only the first step in a new process, which has to be continued by mapping excellence in various other fields. 5. PROCESS AND METHODOLOGICAL ELEMENTS The following process and methodology would be applied to the fields selected: As soon as the identified fields are agreed upon with the HLG for analysis within the pilot study, expert panels in these fields will be set up, thus involving the scientific and industrial communities from a very early stage in the process. The credibility of the exercise would be increased if experts from the MS could be identified by the members of the HLG 4. MS, European science and industry organisations or the Commission services could suggest additional experts, also from non-eu countries. The first role of the expert panels would be to characterise the fields by keywords. The lists of keywords should be broad enough in order to ensure the capture of a large number of scientific publications for the bibliometric screening. The established panels should be large enough to have a broad knowledge of what the current state of art is within the respective national scientific and industrial communities and should also have the means to contact the scientific and industrial organisations, research institutions, and/or further experts. The keywords will fuel a bibliometric analysis, which will be performed by one or more external contractors. In the cases of Life sciences and Nanotechnologies, the database to be used will be the Science Citation Index (SCI) 5 and, whenever appropriate, any other sources of information in the area (e.g. conference proceedings, Websites) where cutting edge research articles are published. For Economics, the Social Science Citation Index (SSCI) is not the database with the widest coverage. However it is the only one enabling citation analysis. Specifically for the screening of hot fields in Economics, it is suggested to use also other databases. 4 It would be opportune to identify experts from research institutions as well as industry. Older scientists have in general a good knowledge of the 'who is doing what' within the research community. However, in order to grasp also new trends and new research groups it is preferable to nominate also young upcoming researchers, for example on a postdoc level. 5 Criticism of this US-American, English-language dominated database is widely known. However, there is no European database available that would enable not only quantitative analysis (number of publications) but also a qualitative analysis (citation analysis). Despite its many shortcomings, the SCI is in particular adequate for analysing the life sciences. The database has a bias towards the life sciences and English is the dominant language in the life sciences world-wide. It is nonetheless a highly recognised database in scientific communities including native English speaking as well as non-native speakers. 7
In all cases, titles and abstracts of articles will be identified if they feature a keyword. Therefore with this, all relevant articles will be identified and, subsequently, authors and their affiliation (an author can have positions in several entities) can be identified 6. This stage will not lead to any exclusion on grounds of size of a country, scientific community, and/or entity but will be based on all publications. For the bibliometric analysis, citation rates should be applied as a proxy for scientific quality. In order to include further indicators, cleaning of data is in some cases necessary. In various fields, publications are not the only important output since entities tend to patent. It will therefore be important to analyse the technological output in those fields where publications alone do not give a complete picture. Another indicator might also be the participation in the Framework Programme. It would indicate some formalised or structured networking. The results of the bibliometric analysis need to be verified and validated. Due to the mobility of researchers, recently established groups, and also due to delays in updating the database, it is possible that some new active groups are missing or that some other groups have changed. These have to be detected through a cross-check with: identified excellent entities from evaluation schemes and prior analyses at national level to be provided by the HLG members; the own knowledge of the experts from the above-mentioned panels. However, if these entities are not merely regrouped and already detected entities under different affiliations, they should not just be added to the identified ones, but checked by bibliometrics and peer-review. The resulting complete lists of scientifically active entities would be made available to the HLG. The next steps will require the setting-up of evaluation panels involving scientific peers, individuals from industry, as well as social scientists. The latter should be well versed in sophisticated evaluation methods like reputation assessment, in order to minimise the negative effects of an ordinary peer-review method. The work to be carried out by these evaluation panels will be based on guidelines established by the Commission in consultation with the HLG. The lists of identified active entities will be limited to the most competent ones by applying cut-off criteria. Any cut-off has to be primarily based on scientific quality and should be taken with reference to the world standard, for example, "above world average". The decision on this threshold will be taken following recommendations of the contractors selected following the calls for tenders (see 3.2) and discussion with the expert panels and the HLG. The resulting list of most competent entities would be disseminated. For a full analysis of the entities above the threshold, the analysis will be enriched with more data 7 on the entities level. This could, for example, be a mission statement, data on personnel (e.g. number of scientists, technical personnel, postdocs, doctoral students) and financing (e.g. percentage of institutional funding, contract money, donations). For longer-term analysis, information about the networking behaviour of these entities would also be relevant. Together 6 For the mapping outcome, the level of the individual researcher would not be revealed; in fact, this will only be part of the working process. 7 At this stage, it is assumed that the entities will be ready to provide the necessary information. 8
with the already available data on scientific and/or technological output, this additional information will be very useful for a balanced evaluation. It allows combining various input and output indicators and enables first appreciation of competencies beside scientific excellence that will be of particular interest for industry and other possible co-operation partners. 8 This will reveal if the entities that scored the highest in the bibliometric citation analysis will equally be on top of the resulting list or if there are other factors leading to a different ranking of the entities. The evaluation outcome will determine which of those entities can be considered excellent. Several evaluation methods could in fact be applied in parallel, but it is of utter importance to present all criteria used in a transparent manner. This step will either reveal a need for additional indicators, methodological amendments, or validate the method. As the first mapping exercise is clearly a pilot for the further analyses, it could be valuable to perform a complementary survey on a limited scale, within the expert and evaluation panels. Each participant in these panels could be asked to list what they think are, say the top 10 research groups in academia and/or industry in his or her country and the top 10 abroad; this without prior knowledge of a bibliometric analysis. The outcome of this small-scale survey could then be crosschecked with the lists coming from the bibliometric analysis. The results could be used to refine these lists. If in addition the experts express a need to enlarge the survey, it could be done, within the limitations mentioned in 3.2. Likewise, a survey from the industries could be performed, focusing, e.g., on questions like top 10 within a country and outside as well as co-operation behaviour. It should be clear that this last exercise would be out of range for the Commission's capacities but the Member States and/or industrial groups may volunteer to organise such a survey. As a first step in disseminating the results of the exercise, data will be presented in an aggregated manner e.g. regrouping entities at regional level. The indicators-based approach will be monitored during the whole exercise by independent expertise. It is also foreseen to assess the impact of the first exercise in a later stage. In addition, since this exercise is the beginning of a process, it is expected to create a network of expertise for running subsequent similar exercises. 6. ROLES OF THE HIGH LEVEL GROUP AND THE COMMISSION 6.1. Role of the HLG The HLG should provide assistance and expertise in the following ways: 1) providing input on national experience and viewpoints; 2) identifying sources of information for data on national RTD entities, to be used in the course of the exercise; 3) assisting in finding appropriate national experts for those tasks requiring expertise; 4) identifying the appropriate intermediaries for information to and feedback from national RTD communities. 8 Data submitted to the evaluation panels will be disseminated only if not considered confidential. 9
Moreover, the HLG (or groups of members) may examine specific issues arising in the course of the implementation of the methodology. In general, the HLG is expected to have the necessary authority to provide support for the mapping exercise to cover each of the EU countries satisfactorily. 6.2. Role of the Commission services For its part, the Commission will 1) liaise with the HLG and provide the necessary technical support to the mapping process like the preparation of the work of the HLG, the involvement of representative European organisations, the transmission of the relevant information to and from the contractors, the co-ordination of the activities; 2) convene all experts necessary for the exercise and co-ordinate their work, including also the organisation of a monitoring activity; 3) assume the costs for the calls for tenders; 4) ensure appropriate dissemination of the results of the exercise. 7. TIMETABLE AND FUTURE DEVELOPMENTS Following the final discussion of the methodology held in the HLG in December 2000, the timetable foreseen to meet the deadline referred to in the Lisbon conclusions and the Council resolution of 15.06.00 would be: - March 2001: Working document from the Commission services transmitted to the Council. - March-May 2001: Selection and definition of precise S&T fields to be mapped, in consultation with the HLG. - July 2001: Launch indicators-based mapping. - September-October 2001: Prepare dissemination activities, and agree a plan for follow-up actions to exploit the expected results. - November-December 2001: Evaluation and validation by panels, following guidelines established in consultation with the HLG. - December 2001: Conclude the first exercise and present its results, together with recommendations for follow-up actions; agree on S&T fields and methodology for the second exercise (see below). Future exercises: The intention is to draw conclusive results from the first exercise, showing that maps are useful tools for decision making. Next exercises should be designed with an aim to: (1) consolidate, and as far as possible simplify, the methodology, reviewing among others the process of field identification and the use made of data from MS and RTD entities; (2) increase substantially the coverage in terms of numbers of fields, so that most of the EUrelevant S&T fields would be mapped by the end of 2002. 10
11