Assessment of progress in achieving ERA in Member States and Associated Countries. Final Report to DG Research and Innovation

Transcription

1 Assessment of progress in achieving ERA in Member States and Associated Countries Final Report to DG Research and Innovation 8 May 2015

2 This page is intentionally blank

3 Assessment of progress in achieving ERA in Member States and Associated Countries Final Report to DG Research and Innovation A report submitted by ICF International in association with Technopolis Date: 8 May 2015 Job Number Elta Smith ICF Consulting Services Limited Watling House 33 Cannon Street London EC4M 5SB T +44 (0) F +44 (0)

4 Document Control Document Title Assessment of progress in achieving ERA in Member States and Associated Countries Job No Prepared by Checked by ICF: Elta Smith, Jan Franke, Andrew Jarvis, Prateek Sureka, Stefania Chirico Technopolis: Viola Peter, Paul Simmonds, Peter Kolarz Elta Smith and Andrew Jarvis Date 8 May 2015 This report is the copyright of DG RTD and has been prepared by ICF Consulting Services Ltd under contract to DG RTD. The contents of this report may not be reproduced in whole or in part, nor passed to any other organisation or person without the specific prior written permission of DG RTD. ICF has used reasonable skill and care in checking the accuracy and completeness of information supplied by the client or third parties in the course of this project under which the report was produced. ICF is however unable to warrant either the accuracy or completeness of such information supplied by the client or third parties, nor that it is fit for any purpose. ICF does not accept responsibility for any legal, commercial or other consequences that may arise directly or indirectly as a result of the use by ICF of inaccurate or incomplete information supplied by the client or third parties in the course of this project or its inclusion in this project or its inclusion in this report. Disclaimer: The views expressed in this report, as well as the information included in it, do not necessarily reflect the opinion or position of the European Commission and in no way commit the institution. 2 April 2015 i

5 Contents Executive summary... i ES1.1 ES1.2 ES1.3 ES1.4 ES1.5 ES1.6 Introduction...i The European Research Area...i Study scope and context... ii Indicators and suitable metrics to measure progress... ii Roadmaps and peer review approaches to measure progress... iv A future evaluation and monitoring system...v 1 Introduction Study scope and objectives Report structure An ERA framework ERA background and objectives The European Research Area and the Innovation Union Flagship Initiative Rationale and intervention logic of ERA priority areas Indicator identification and appraisal Approach of the appraisal Priority 1 - indicator assessment Priority 2 - indicator assessment Priority 3 - indicator assessment Priority 4 - indicator assessment Priority 5 - indicator assessment International Dimension Outside ERA (Priority 6) - indicator assessment Indicator selection - summary The role of national roadmaps in achieving the ERA Political context at European level and steps towards a European roadmap State of play in Member States Potential role of national roadmaps in the ERA monitoring framework Role of peer reviews in ERA An introduction to peer review Previous peer reviews in the context of the ERA Use of peer review in other policy areas Conclusions and recommendations Indicators and suitable metrics to measure progress Desirable attributes of a future evaluation and monitoring system Towards an integrated approach to measuring ERA progress An illustrative rating system for progress reporting Annex 1 Study tasks and work completed against objectives Annex 2 Methodology to assess ERA progress Annex 3 Outcome of scope test Annex 4 Comparable indicators and Annex 5 Indicator data availability 2013 and Annex 6 Summary of evaluation progress, steps Annex 7 ERA intervention logics April 2015

6 Annex 8 Indicator maps and indicator appraisal tables Annex 9 The German ERA roadmap and a template for national roadmaps Annex 10 Overview of ERA peer reviews performed by CREST/ERAC Annex 11 Case studies Peer review and mutual learning mechanisms Annex 12 References Table of tables Table ES1.1 Selected input, output and outcome/impact indicators by priority area... ii Table ES1.2 Traffic light rating system for ERA progress...v Table 3.1 Priority 1 proposed indicators Table 3.2 Priority 2 proposed indicators Table 3.3 Priority 2 proposed indicators along sub-categories Table 3.4 Priority 3 proposed indicators Table 3.5 Priority 4 proposed indicators Table 3.6 Priority 5 proposed indicators Table 3.7 Priority 5 proposed indicators along sub-categories Table 3.8 International dimension outside the ERA (Priority 6) proposed indicators Table 3.9 Selected input, output and outcome/impact indicators by priority area Table 3.10 Indicator selection one indicator per priority area Table 6.1 Traffic light rating system for ERA progress Table A1.1 Study tasks and work completed against study objectives Table A3.1 Priority 1 comparison of progress indicators and data sources Table A3.2 Priority 2 comparison of progress indicators and data sources Table A3.3 Priority 3 comparison of progress indicators and data sources Table A3.4 Priority 4 comparison of progress indicators and data sources Table A3.5 Priority 5 comparison of progress indicators and data sources Table A3.6 International dimension outside ERA - comparison of progress indicators and data sources Table A4.1 Indicators that remain unchanged between 2013 and Table A5.1 Priority 1 - Data availability for indicators that changed between 2013 and Table A5.2 Priority 2 - Data availability for indicators that changed between 2013 and Table A5.3 Priority 3 - Data availability for indicators that changed between 2013 and Table A5.4 Priority 4 - Data availability for indicators that changed between 2013 and Table A5.5 Priority 5 - Data availability for indicators that changed between 2013 and Table A6.1 Priority 1 - Evaluation status Table A6.2 Priority 2 - Evaluation status Table A6.3 Priority 3 - Evaluation status Table A6.4 Priority 4 - Evaluation status Table A6.5 Priority 5 - Evaluation status Table A6.6 International Dimension outside ERA - Evaluation status Table A8.1 Indicator maps - key Table A8.2 Indicator scoring system April 2015

7 Table A8.3 Indicators Appraisal - Priority Area Table A8.4 Priority 1 - appraisal of indicators suggested by DG RTD Table A8.5 Indicators Appraisal - Priority Area Table A8.6 Priority 2 - appraisal of indicators suggested by DG RTD Table A8.7 Indicators Appraisal - Priority Area Table A8.8 Priority 3 - appraisal of indicators suggested by DG RTD Table A8.9 Indicators Appraisal - Priority Area Table A8.10 Priority 4 - appraisal of indicators suggested by DG RTD Table A8.11 Indicators Appraisal - Priority Area Table A8.12 Priority 5 - appraisal of indicators suggested by DG RTD Table A8.13 Indicators Appraisal International dimension Table A8.14 International dimension outside the ERA - appraisal of indicators suggested by DG RTD Table A9.1 Overview of German guidelines and roadmap on ERA Table A10.1 Overview of ERA peer reviews conducted by the European Union Scientific and Technical Research Committee (CREST) and ERAC Table of figures Figure 5.1 Country-to-country peer review process in ERAC using the Innovation Union Self- Assessment Tool Figure A2.1 Flow Diagram for ERA Evaluation Figure A2.2 Summary of the indicator appraisal following the application of steps Figure A2.3 Example of a scorecard Figure A7.1 Intervention Logic ERA Priority 1 More effective national research systems Figure A7.2 Intervention Logic ERA priority 2 Optimal transnational cooperation and competition Figure A7.3 Intervention Logic ERA priority 3 Open Labour Market for researchers Figure A7.4 Intervention logic - ERA priority 4 Gender equality and gender mainstreaming in research Figure A7.5 Intervention logic - ERA priority 5 Optimal access to and circulation and transfer of scientific knowledge Figure A7.6 Intervention logic - ERA priority International dimension outside ERA Figure A8.1 Priority 1 potential input, output and outcome/impact indicators Figure A8.2 Priority 2 potential input, output and outcome/impact indicators Figure A8.3 Priority 3 potential input, output and outcome/impact indicators Figure A8.4 Priority 4 potential input, output and outcome/impact indicators Figure A8.5 Priority 5 potential input, output and outcome/impact indicators Figure A8.6 International dimension outside ERA (Priority 6) potential input, output and outcome/impact indicators Figure A9.1 Draft template for national roadmaps on ERA April 2015

8 Executive summary ES1.1 Introduction This is the final report for a study to assess progress in achieving the European Research Area (ERA) in Member States and Associated Countries. The call for tender was issued by the Directorate- General for Research and Innovation (DG RTD) under a framework contract for the evaluation of research and innovation programmes and policies between ICF Consulting Services, Ltd and DG RTD. ES1.2 The European Research Area The ERA is defined as a unified research area which enables the free circulation of researchers, scientific knowledge and technology. It should enable Members States and the European Union (EU) overall to strengthen its scientific and technological bases, competitiveness and capacity to address grand challenges. There are five ERA priorities plus a crosscutting focus on international cooperation which at ERAC/Member State level is often considered to be the sixth ERA priority. The priorities are: 1. More effective national research systems to increase competition and excellence within national borders and ensure the most efficient and effective funding allocation. ERA priority 1 sets out two main fields of action to tackle structural differences between Member States research and innovation systems and improve value for money, quality and quantity of research outputs. First, Member States, together with relevant national actors, are expected to mainstream competitive project-based funding based on international peer review standards. Second, Member States are expected to design or amend legal measures which govern institutional research funding to introduce institutional funding based on performance and quality of research. 2. Optimal transnational co-operation and competition to establish a common research agenda, improve interoperability of national programmes, and build effective pan-european research infrastructures. One aspect of ERA priority 2 is improvement of the framework conditions for joint programming and transnational cooperation both on funding programmes for research as well as major research infrastructures. 3. An open labour market for researchers to increase researcher mobility, training and attractive career development. This priority area seeks to improve framework conditions for researcher mobility across Europe, across different stages of their career and between the academic and private sector. This includes actions to improve the attractiveness of research careers, structured doctoral training programmes and standards in recruitment of academic staff. 4. Gender equality and gender mainstreaming in research to make better use of diverse scientific human resources as a way to foster quality and relevance of research. Firstly, this priority includes actions to promote gender equality in research and emphasise cultural and institutional changes to remove legal and other barriers to recruitment, retention and career progression of female researchers. It also includes actions to ensure a gender-balanced approach to decision making processes with regards to allocation of funding and recruitment. Secondly, it integrates the gender dimension in research content, programmes and projects. 5. Optimal access to and circulation and transfer of scientific knowledge, including via digital ERA to improve access to and uptake of knowledge transfer and facilitate open innovation. This includes improving open access to publications and access to data resulting from publicly funded research and strengthening the connections between science and industry and the role of publicsector research in open innovation. The crosscutting focus on international cooperation encourages Member States to foster openness for international cooperation to maximise EU research potential. Most Member States already have national strategies for international cooperation on research and innovation (EC, 2014). But national level initiatives could be strengthened through greater coordination between Member States. 2 April 2015 i

9 The Commission has identified actions for each priority area which are expected to be implemented through initiatives and measures undertaken by the Commission, Member States and research stakeholder organisations, acting in partnership. ES1.3 Study scope and context Progress reports and other research have found that progress in achieving ERA in the Member States and Associated Countries has been limited but there have been improvements across most ERA priority areas (Dinges et al., 2013; ERA Progress Report, 2014). These changes have not been uniform across the EU-28, however, with greater progress achieved in Nordic and Western European countries compared to Central and Eastern European countries. As a consequence, the knowledge gap has widened. The present study was expected to examine whether the measures in place in Member States and Associated Countries had advanced since 2012 and were better aligned with the ERA priorities. The terms of reference set the following objectives: update the state-of-play of ERA; develop and estimate policy progress indicators; and evaluate ERA progress in the Member States and Associated Countries. The evaluation was intended to inform the preparation of the second ERA Progress Report to be published in September 2014, but delays in the procurement procedure meant that the present study could not be completed in time to fulfil this original objective. Ongoing work by DG RTD and the Member States and Associated Countries provided some of the information asked for in the original study terms of reference. As a result, the study team supported DG RTD in the preparation of its 2014 progress report and undertook to complete the evaluation of progress. Following preliminary work on the evaluation, it became clear that the evaluation as originally programmed could not be performed and an alternative strategy was agreed with DG RTD to develop an approach for future ERA progress monitoring and evaluation. The objective of the work was to develop a set of proposals for further development of the ERA progress monitoring framework to be considered by DG RTD. ES1.4 Indicators and suitable metrics to measure progress For each priority area, the study team conducted an appraisal of available or potential indicators. The output of this exercise was an inventory of indicators, organised by ERA priority, with an appraisal of their suitability for monitoring and evaluation, and a final suite of indicators proposed as a core set to measure progress across ERA priorities. The final indicator suite is summarised in Table ES1.1. Table ES1.1 Selected input, output and outcome/impact indicators by priority area 1 Priority Input Indicator Output Indicator Outcome/ Impact Indicator Priority 1: more effective national research systems 2a ESFRI 2b Transnational cooperation Total GBAORD 2 as per cent GDP (OECD) Per cent of MS participating in the development of at least one of the RIs identified by ESFRI Degree to which MS engage in transnational cooperation via an EU framework programme Share of national Number of patents per total GBOARD allocated public research and through project-based development (R&D) funding (as opposed to expenditure (Eurostat) institutional funding) (ERA Survey) Per cent of research performers experiencing problems accessing RIs Share of public funding allocated to transnational R&D cooperation Number of implementation phase ESFRI projects in which each MS is a partner Cross-border ownership of patents 1 Data sources are indicated in brackets. 2 Government budget appropriations or outlays for research and development 2 April 2015 ii

10 Priority Input Indicator Output Indicator Outcome/ Impact Indicator Priority 3: Open labour market for researchers Priority 4: Gender equality and gender mainstreaming in research Per cent of research funding available for mobility scholarships and stipends of the total funding for research (MORE2, JRC) Proportion of female PhD (ISCED 6) graduates (She Figures, based on Eurostat) 5a Open Access Share of funders funding open access to publications (ERA Survey) 5b transfer Knowledge Share of organisations that has or uses a structure for knowledge transfer activities (ERA Survey) International Share of the public R&D Non-EU dimension outside budget allocated to ERA (Priority 6) collaborative programmes with third countries (ERA Survey) Share of research organisations with EC Human Resources Excellence in Research Acknowledgement (EC web site, JRC) Proportion of female academic staff (She Figures, based on data from Women in Science (WiS) database) (Data collection needed) R&D in higher education institutions (HEIs) / public research organisations (PROs) funded by business (Eurostat) doctorate holders as a per cent of total doctorate holders (Eurostat via the IU Scoreboard) Proportion of doctoral candidates with a citizenship of another EU MS (Innovation Union (IU) Scoreboard, pending) Alternative: Researchers working in the business sector (Eurostat) Proportion of women in grade A (professor) positions (She Figures) (Data collection needed) Public / private co-publication per million of the population Licence and patent revenues from abroad as a per cent of GDP (Eurostat via the IU Scoreboard) The advantages and limitations of reducing the indicator suite to a set of six indicators, one for each priority area, were assessed. It is possible in principle to reduce the three selected indicators per priority to only one. To ensure comparability among the priority areas, these should be either the six inputs, outputs or outcome/impact indicators. The outcome / impact indicators are recommended for monitoring ERA progress because a lack of progress on any of them would suggest that ERA-related policies and initiatives may not be having their desired effect and warrant further investigation. Though this approach is straightforward and user-friendly, such a minimalist selection of indicators is problematic because it is difficult to represent progress on what are very broad ambitions by reference to a single, narrow indicator. For example, selecting input, output and outcome/impact indicators for each priority area can provide insight into whether resources have been invested in each priority, whether benefits are observable, and whether there is evidence of wider impacts, resulting in an overall ability to assess effectiveness in fulfilling ERA priorities. Composite indicators could provide an aggregated view on progress, but current variation in data availability and relevance of available indicators highlighted in this study suggest this is not feasible at present. Furthermore, the indicator appraisal highlighted data collection needs and opportunities for strengthening the evidence base for relevant indicators in each ERA priority. Nevertheless, the analysis conducted of indicators across ERA priorities demonstrated that existing data sources provide information that can be used for ERA progress monitoring. One of the main issues with current data from ERA surveys is the variation in response rates, which should be addressed in future survey design. There are gaps in some other areas as well; these do not suggest the need for entirely new data collection exercises but rather a need for more complete and comprehensive data from existing sources. 2 April 2015 iii

11 ES1.5 Roadmaps and peer review approaches to measure progress Member States vary in their levels of ERA attainment, ambitions, and resources, and in the research systems scale and structure. National roadmaps that set country-specific paths to achieving the ERA, and which define milestones and timelines, have a potentially important role in an ERA monitoring framework. This study provided an assessment of that role and how roadmaps and peer review approaches could complement other components of the ERA monitoring system. The analysis suggests that for national roadmaps to provide added value to existing monitoring activities, they should have a common structure whilst allowing Member States flexibility in pace and scope of action. Advice and guidance on the structure could be provided as part of the upcoming European roadmap on ERA, which is currently being prepared by the European Research Area and Innovation Committee (ERAC) in cooperation with the European Commission, and expected to be presented in the first half of The research undertaken for this study found that ERAC and the Commission should aim to provide certain framework conditions in developing guidance and advice on national roadmaps as part of the European ERA roadmap. This would include a set of non-binding recommendations and aim to establish a high level of mutual trust between Member States and the Commission through clear and transparent discussion on, and review of, national roadmaps within the context of ERAC. To support the development of a progress monitoring framework that includes national targets and pathways to completion of the ERA, the national roadmaps should: clearly identify relevant actors within the national research system and their role in achieving progress; include a clear definition of specific problem areas and challenges which inhibit a Member State in its efforts to implement the ERA; define quantitative targets and the actors responsible for meeting them; give a timeframe for achievement of individual activities and targets; explain the mechanism to be used for progress reporting, including the actors responsible for progress reports; and commit to review progress on a regular basis, describing the actors that need to be involved and planned frequency of review. These requirements could be set out in the European roadmap. Furthermore, past experience of peer review in the ERA and the case study examples prepared for this study suggest that a peer review mechanism could have a positive role to play in completing the ERA if appropriately constructed, operated and resourced. Future ERA peer reviews could be organised as a joint effort by the Commission and Member States, built on the concept of shared responsibility (TFEU Article 182.5). Member States can be encouraged to initiate development of a common approach and guidelines to establish a formalised and credible peer review system. The planned Policy Support Facility could serve as a hub for expertise and provide administrative support and guidance to external experts and peer review participants. The peer review mechanism s prospects for success would be improved if it incorporated the following elements and approaches: decisions on the peer review programme and approval of the outputs being taken by ERAC; a properly documented peer review process that was well understood by Member States; the scope of reviews and selection of reviewers was organised based on the principal ERA objectives or individual measures identified in the EU ERA roadmap in 2015; a structured approach to provide for national roadmaps and a corresponding EU feedback mechanism, possibly linking into the European Semester; 2 April 2015 iv

12 the peer review teams including a mix of experts, including independent experts, rather than being dominated by Member State policy administrators; a structured approach towards the provision of thematic and administrative support, and the development of guidelines, templates and a structured knowledge management system for collected data and analytical reports, that encouraged more Member States to express an interest in being peer-reviewed and reduced administrative burden on national representatives; an annual summary of peer reviews, drafted by ERAC (i.e. by the Member States themselves), that strengthened ownership, provide guidance for the EU as a whole and built mutual accountability between Member States; and a review of the Innovation Union Self-Assessment Tool (SAT) and the inclusion of relevant overlapping policy fields to increase interest from Member States. The SAT currently forms the basis and thematic framework for peer reviews in ERAC, but might neglect or miss essential topics and issues that are addressed within the national research and innovation system. ES1.6 A future evaluation and monitoring system Current monitoring arrangements and the indicators identified by the study team only cover parts of the overall activities. Complementary approaches are required to reflect Member State diversity and encourage progress reporting. Building on the work already undertaken by ERAC, any future integrated approach should be built around a core set of indicators with individual national roadmaps. This arrangement should allow for specific focus on strengths and weaknesses of national systems and iterative performance review against a core set of indicators, accompanied by individual national objectives set out in roadmaps. This could also strengthen the political visibility of ERA in national research governance systems. There is also limited integration and coordination between the ERA and complementary European strategies on innovation and economic growth. The Innovation Union s Self-Assessment Tool has been used in a series of ERA peer reviews. A number of indicators from the Innovation Union Scoreboard are being used by the ERA progress reports. But while the Innovation Union groups Member States according to structural similarities and performance using a scoreboard approach, there is no structured performance measurement in ERA which would allow for a similar grouping of countries according to ERA priorities. A more structure approach could strengthen the political dimension and visibility of ERA. Progress and performance at EU level could be demonstrated across ERA priorities through the use of a traffic light system. This could signal the status of progress towards ERA objectives and EU level goals. Progress could be assessed against the goals defined under the EU roadmap. Progress could be assessed against all ERA priorities or specific areas of action identified in the roadmap. An example of how this approach could be used in the ERA is provided in Table ES1.2. Table ES1.2 Traffic light rating system for ERA progress Performance Description Taking all relevant information into account, the aggregate assessment is that this aspect of the ERA is either complete or track for completion by the given deadline. The available information suggests that achievement of this aspect of the ERA within the given timetable is in doubt. There are issues to be addressed but the situation is not irrecoverable providing the risks are addressed. There are significant problems and achievement of this aspect of the ERA is not expected within the given timetable. Progress objectives are not being met and actions is required. Status This approach offers succinct and easily identifiable messages about the state-of-play at EU level on ERA priorities. Traffic light ratings could be assigned by a team of individual experts, who review data collected against the core indicator set and peer review reports. 2 April 2015 v

13 1 Introduction This is the final report for a study to assess progress in achieving the European Research Area (ERA) in Member States and Associated Countries. The call for tender was issued by the Directorate-General for Research and Innovation (DG RTD) under a framework contract for the evaluation of research and innovation programmes and policies between ICF Consulting Services, Ltd and DG RTD. The study was delivered by a team led by ICF with the support of Technopolis, Lancaster University and Delft University of Technology. 1.1 Study scope and objectives The ERA is central to the Europe 2020 strategy and its Innovation Union (IU) initiative. The following ERA definition is presented in the European Commission (EC) ERA Communication (EC, 2012d) and is based on the Lisbon Treaty 3 and European Council conclusions: A unified research area open to the world based on the internal market, in which researchers, scientific knowledge and technology circulate freely and through which the union and its Member States strengthen their scientific and technological bases, their competitiveness and their capacity to collectively address grand challenges. The Commission defined five priority areas and a cross-cutting focus on international cooperation to achieve the ERA as envisioned (EC, 2012a and 2012d): 1. more effective national research systems to increase competition and excellence within national borders and ensure the best and most effective funding allocation; 2. optimal transnational co-operation and competition to establish a common research agenda, improve interoperability and comparability of national programmes, and build effective pan-european research infrastructures; 3. an open labour market for researchers to increase researcher mobility, training and attractive career development; 4. gender equality and gender mainstreaming in research to make better use of diverse scientific human resources as a way to foster quality and relevance of research; and 5. optimal access to and circulation and transfer of scientific knowledge to improve access to and uptake of knowledge and facilitate innovation. The Commission has also identified actions for each priority area expected to be implemented through initiatives and measures undertaken by the Commission, Member States and research stakeholder organisations, acting in partnership (EC, 2012d). Dinges et al. (2013) observed that following the ERA Communication (EC, 2012d), progress in achieving ERA in the Member States and Associated Countries was limited: the majority of countries reviewed had only implemented half of the indicators to a medium degree. The ERA progress report 2014 (EC, 2014a) suggests that there has been progress across most of the ERA priority areas. Developments across the European Union (EU) were not uniform, however, with greater progress achieved in Nordic and Western European countries compared to Central and Eastern European countries. Southern European countries were mixed in this respect. 3 Article 179 of the Consolidated Version of the Treaty on the functioning of the European Union (EU, 2012). 2 April

14 The present study was expected to examine whether the measures in place in Member States and Associated Countries 4 had advanced since 2012 and were better aligned with the ERA priorities. The terms of reference set the following objectives: update the state-of-play of ERA; develop and estimate policy progress indicators; and evaluate ERA progress in the Member States and Associated Countries. In the period between the submission of ICF s proposal and the start of the contract the Commission developed and estimated the policy progress indicators, Member States submitted updates of the National Reform Programmes (NRP), and surveys of public research organisations and research funding organisations were launched. The surveys were expected to provide data on ERA monitoring indicators, which were to a large extent included in the ERA impact assessment report (EC, 2012b) and agreed with the Member States. The evaluation was intended to inform the preparation of the second ERA Progress Report, to be published in September 2014, but delays in the procurement procedure meant that the present study could not be completed in time to fulfil this original objective. On-going work by DG RTD and the Member States and Associated Countries provided some of the information asked for in the original study terms of reference. As a result, the study team supported DG RTD in the preparation of its 2014 progress report and undertook to complete the evaluation of progress (the tasks completed are summarised in Annex 1). Following preliminary work on the evaluation, described and documented in Annex 2 through Annex 6, it became clear that the evaluation as originally programmed could not be performed, and an alternative strategy was agreed with DG RTD to develop an approach for future ERA progress monitoring and evaluation. The objective of the work was to develop a set of proposals for further development of the ERA progress monitoring framework to be considered by DG RTD. The results of this work programme are set out in this report. 1.2 Report structure The remainder of this final report is structured as follows: Section 2 sets out a framework explaining the ERA ambition and expected mechanisms of change; Section 3 identifies indicators that could be used to measure ERA progress and appraises their suitability; Section 4 assesses the potential role of national roadmaps in achieving the ERA; Section 5 considers potential approaches to support monitoring and performance management mechanisms in the context of ERA; and Section 6 provides recommendations on system development to assist DG RTD in the future evolution of the ERA. 4 Namely: Albania, Bosnia and Herzegovina, Faroe Islands, Macedonia, Iceland, Israel, Lichtenstein, Moldova, Montenegro, Norway, Serbia, Switzerland and Turkey (EC, n.d.). 2 April

15 2 An ERA framework Monitoring and evaluation must be set in the context of the ambitions identified for the ERA and mechanisms by which change is expected to occur. This includes having a clear understanding of the ERA objectives, its activities, outputs and outcomes, as well as the expected impacts. This section sets out the rationale for the ERA, and then for each of the priority areas through intervention logics. An intervention logic is an analytical tool that shows how priorities are operationalised, illustrating the path from objectives to expected results (impacts). Intervention logics have been prepared for each priority area, illustrating the mechanism(s) by which inputs (such as research funding and infrastructure) are connected to outcomes (and achievement of the strategic objectives) via activities and outputs. These serve as a model of how the system should work. Figures illustrating the intervention logics for each priority area are provided in Annex 7. They help demonstrate where the current progress monitoring mechanism is focused and where gaps can be identified. They illustrate the main actions, inputs, outputs, outcomes and expected impacts for each priority area. The intervention logics are complemented by a set of indicator maps identifying potential indicators against the proposed intervention logic. The proposed indicators are presented in section ERA background and objectives The ERA was conceptualised as an instrument to integrate research resources and capacity across EU Member States, mirroring the common market. The ERA was introduced to support the Lisbon Agenda, which set out the EU s strategic economic development goals (European Council, 2000). The Lisbon Treaty and its amendments established research policy as a shared competence between the European Commission and the Member States, reinforcing the community dimension of research policy and providing a legal basis for EU action on ERA. 5 The ERA Vision (EC, 2000) and the ERA Green Paper (EC, 2007) identified fragmentation and compartmentalisation of national research efforts as major issues to address, and as fundamental to the rationale for a unified research area. The Commission s objectives included strengthening the EU s global competitiveness and eliminating the EU s innovation gap with the US and Japan. The Innovation Union Scoreboard 2014 suggests that this gap is closing slowly, but the US continues to be ahead of the EU in the commercial exploitation of research results. EU investments in research and development (R&D) in relation to the gross domestic product (GDP) are below the levels of competitors. The ERA contributes to improvements in these areas by tackling major differences in innovation and research performance between Member States, particularly the uneven spread of knowledge production and innovation (EC, 2014). Major trends also affect the EU s social and economic development, and impact on its innovation and research systems. A joint EU approach which is founded on coordinated action in the field of research and innovation is considered to be the best way to address these challenges including climate, energy and resource scarcity; security concerns and emerging conflicts; and the rise of a service and knowledge-based economy (EC, 2008). 5 Article of the Consolidated Version of the Treaty on the Functioning of the European Union (TFEU) (EU, 2012). 2 April

16 2.2 The European Research Area and the Innovation Union Flagship Initiative The ERA operates alongside the EU s strategic growth agenda. Europe 2020 and the Innovation Union flagship initiative address framework conditions and access to finance to enable exploitation of research and innovation in products and services (EC, 2010). The Europe 2020 strategy includes specific development targets, including a target to spend three per cent of the EU s GDP on R&D by The Innovation Union, announced as one of seven flagship initiatives in the Europe 2020 strategy, is intended to improve the framework conditions for research and innovation in Europe, including ERA completion. The Innovation Union aims at creating an internal market for innovation. The initiative has therefore set out the following objectives in 2010 (EC, 2010): Member States should leverage investment in education, R&D, innovation and information and communication technologies (ICTs); EU and national research and innovation systems should be better connected; education systems should be modernised and focused on excellence; the ERA should be completed by 2014; access to EU funding programmes for research and innovation should be simplified and their leverage effect on private sector investment, i.e. the amount of private investment triggered by public funding, must be enhanced; cooperation between science and business should be enhanced to enable more effective commercial exploitation of research; European Innovation Partnerships should be launched to accelerate research, development and market deployment of innovation for major societal challenges; strengths and potential in design and creativity should be better exploited; and international cooperation in R&D should be improved. The Commission provides three main instruments to measure progress against these targets: a self-assessment tool for Member States to review their national and regional research and innovation systems; a regular review of performance against the objectives listed above using a performance scoreboard approach (i.e. the Innovation Union Scoreboard); and European Innovation Partnerships, which bring together aspects of R&D and market deployment along thematic areas of societal concern (e.g. health, agricultural sustainability, smart cities and communities, water, and raw materials). There has been limited integration and coordination between the Innovation Union initiative and ERA activities, despite thematic overlap. The Innovation Union s self-assessment tool has been used in a series of ERA peer reviews. Knowledge management and data analysis systems are available through the ERAWATCH portal and the Innovation Union Dashboard. A number of indicators from the Innovation Union Scoreboard are being used by the ERA progress reports. But while the Innovation Union groups Member States according to structural similarities and performance using a scoreboard approach, there is no structured performance measurement in ERA which would allow for a similar grouping of countries according to ERA priorities. 2 April

17 2.3 Rationale and intervention logic of ERA priority areas Five priority areas for ERA action were identified in the 2012 Communication on 'A Reinforced European Research Area Partnership for Excellence and Growth' (EC, 2012): 6 The following sections outline detailed activities, inputs, direct outputs, outcomes and wider impacts for each priority area ERA priority 1 - more effective national research systems Almost all Member States have adopted a national strategy on research and innovation (EC 2014). The Innovation Union Scoreboard 2014 suggests that there are wide performance gaps between innovation leaders and modest innovators who lag behind. Whilst respecting the specifics of national research systems, structural differences and variation in institutional set-up as well as different approaches to allocation of funding are a significant structural problem preventing national research systems from becoming more integrated, competitive and effective. ERA priority 1 sets out two main fields of actions to tackle these structural differences and improve value for money, quality and quantity of research outputs. Under this priority Member States are expected to establish, maintain and develop performance-enhancing structures, framework conditions and processes such as national strategies, specific funding programmes adhering to these standards, and organisational change in research funding organisations (RFOs) to reflect these priorities. First, under Action 1a, Member States, together with relevant national actors such as RFOs and programme management agencies, are expected to mainstream competitive projectbased funding based on international peer review standards. This may happen, for example, through specific national strategies focussing on competitive funding of projects or clusters of projects that have been peer reviewed by domestic or international experts. National strategies might focus on specific research areas to build upon existing national infrastructure, scientific expertise or existing industries to commercialise results. Examples include the German High-Tech Strategy and Excellence Initiative, which is expected to increase the share of competitively allocated research and development funding and increase the share of peer-reviewed projects in total research and development spending. Under Action 1b, Member States are expected to design or amend legal measures that govern institutional research funding and, through RFOs and individual measures, govern institutional funding mechanisms, introducing qualitative performance goals without compromising long-term financial planning certainty. These activities will directly result in an increased share of institutional funding allocated to research performing organisations (RPOs) based on quality-oriented performance measurement and/or on the evaluation and appraisal of performance-related indicators. A direct outcome of these activities should be a stronger focus on scientific and technical excellence in allocating national research and development funding, an increased number of high-impact publications and increased social and commercial impact of research projects. Long-term impacts of the activities, outputs and outcomes outlined above may include improved capacity and efficiency of national research systems and allow for a higher degree of regional specialisation, enable better performance in overall scientific and commercial output and reduce unintentional overlap in RPO research profiles ERA priority 2 optimal transnational cooperation and competition ERA priority 2 focuses on, amongst other aspects, improving the framework conditions for joint programming and transnational cooperation both on funding programmes for research, as well as major infrastructures. Most transnational cooperation within the EU is 6 Based on the ERA Green Paper (EC, 2007). 2 April

18 implemented via European framework programmes and activities of the European Space Agency. Evaluation of the EU s framework programmes and schemes such as ERA-NET 7 show that these bring about a strong economic impact and structure EU research efforts towards more efficient mainstreaming and capacity building of research agendas (EC, 2012b). Member State interest in Joint Programming Initiatives (JPIs) and support to ERA-NET Co-fund actions, as well as Article 185 initiatives in Horizon 2020 all indicate further development towards increased and more comprehensive transnational cooperation. But Member States still have significant hurdles to overcome. For example, some Member States are working on national action plans, roadmaps and strategies to further develop JPIs and align national research agendas with initiatives supported under Horizon Twelve Member States have made provisions to foster bilateral or multilateral international cooperation (EC, 2014). Under the ERA, Member States are expected to improve framework conditions for JPIs with a specific focus on grand challenges. Member States and RFOs should also continue to remove legal and administrative barriers to ensure evaluation practice and funding rules conform to international standards and are compatible across Europe. These activities could result in short-term outputs such as increased national budgets for JPIs and other transnational cooperation instruments, as well as an increased share of funding allocated along compatible and interoperable evaluation practice. In the mediumterm, these outputs could increase EU research capacity and allow for a more coordinated approach and critical mass of resources to: leverage additional public and private investments in research; increase the average impact of co-authored work; and allow for wider knowledge dissemination and spillover effects across the European economy. The long-term impact of a JPI could include contributions to solving grand societal challenges of cross-border relevance through research, and aligning national strategies in the selected JPI domains. The second main focus of ERA priority 2 is to improve the capacity and development of, as well as access to, large national and pan-european research infrastructures. Improved research infrastructures can improve the scale and speed of major research undertakings. Cross-border access to national infrastructure and the development of pan-european infrastructures also offer financial benefits as well as distributed costs of development, maintenance and staffing, which may be shared across Member States. Twenty-two Member States have adopted national roadmaps on research infrastructures, although many of them do not show consistent links with EU-level efforts and financial commitments to establish infrastructures of pan-european interest identified by the European Strategy Forum on research Infrastructures (ESFRI) (EC, 2014). Member States are therefore expected to reserve budgets for the preparation, development and maintenance of ESFRI roadmap projects. They should also develop their national research infrastructure strategies to remove legal, technical and other barriers to enable complete cross-border access to national research infrastructure. This is expected to result in improved access to national infrastructures by non-nationals and improved financial security, financial capacity and bundled capacity for the development and maintenance of research infrastructures including completion of ESFRI roadmap projects. In the long-term, resources for, and access to both national infrastructures and ESFRI roadmap projects should be pooled and the development of new infrastructures coordinated transnationally. 7 The ERA-NET (networking) scheme was launched in 2003 to support networking activities leading to improved cooperation and coordination of national and regional research programmes carried out by Member States and Associated Countries (EC, 2003). 2 April

19 2.3.3 ERA priority 3 open labour market for researchers Free movement of knowledge has been highlighted as the EU s fifth freedom needed to maintain a competitive and attractive EU labour market, a knowledge-driven economy and to avoid brain-drain through the loss of European talent to competing regions such as Brazil, Russia, India, China (BRIC countries) and the US. EU programmes like the Marie- Skłodowska-Curie instruments contribute to the mobility of EU researchers and international research, but important barriers remain at Member State and EU level regarding the working conditions and mobility of researchers. These include variety in transparent and fair recruiting of research staff, low levels of staffing autonomy in many research performing organisations, varying and incompatible career structures across the EU as well as legal, administrative and language barriers for non-national and third-country staff. In a number of Member States an open, transparent and merit-based recruitment system is not in place, intersectoral mobility is low and working conditions as well as the overall attractiveness of scientific careers are insufficient (EC, 2014a). ERA priority 3 sets out a number of actions to tackle these problems. These include Member States activities to introduce or expand structured doctoral training, programmes to increase mobility between industry and academia, and efforts to remove barriers to cross-border portability of national grants. RFOs are expected to implement and adopt the EU s European Code of Conduct for the recruitment of researchers and oblige funded institutions to comply with the European Charter of Researchers and Code of Conduct for the Recruitment of Researchers. The Principles of Innovative Doctoral Training (IDTP) defined by the ERA Steering Group on Human Resources and Mobility (ERA SGHRM, 2013) should be adopted by RFOs, which is expected to improve cross-sector mobility of researchers. The Commission is expected to further improve the usefulness, usability and usage of the EURAXESS portal 8 and support the establishment of a transnational pension fund (RESAVER 9 ) for research organisations and their employees (EC, n.d.b). This should result in further improvements to the openness and fairness of recruitment procedures, improved working conditions and attractiveness of research careers, and increased mobility of researchers internationally and across sectors. In the long-term, activities under this ERA priority are expected to help strengthen the EU workforce by attracting more people to research careers and providing both academia and industry with better trained personnel ERA priority 4 gender equality and gender mainstreaming in research Gender equality and mainstreaming have gained increased recognition on policy agendas at national, European and international levels, as well as within research organisations including universities. Significant steps have been undertaken under ERA priority 4 on gender issues in research and innovation. But skilled female research capabilities are underutilised and women are under-represented across career paths. This may undermine the quality and relevance of research outputs and represents an inefficient use of talent across the EU (EC, 2012). Priority 4 objectives include improving gender equality and strengthening the gender dimension in research programmes. The ERA Communication (EC, 2012) invited Member States to create a legal and policy framework to promote and incentivise gender equality. Specific national policies on gender equality in public research have been adopted in 17 countries. According to the ERA progress report 2014, the proportion of women in recruitment committees and evaluation panels was 36.6 per cent and 35.8 per cent respectively compared to the target of 40 per cent set by the Commission in the Communication (EC, 2014c). 8 The EURAXESS Researchers in Motion jobs portal provides recruitment support services to researchers with the aim of improving researchers mobility in the EU (EC, 2015b). 9 More details of RESAVER are provided online at 2 April

20 Action 4a refers to gender equality in research and emphasises cultural and institutional changes to remove legal and other barriers to recruitment, retention and career progression of female researchers. Member States are expected to design national policies on gender equality in public research. RPOs are asked to adopt Gender Equality Plans and implement changes in their recruitment and promotion policies to off-set current imbalances. This would be expected to result in more gender-balanced recruitment across RPOs. Member States are expected to ensure that there is a gender balance in decision making processes with regards to allocation of funding and recruitment (Action 4b). The Commission has invited Member States to ensure that committees which are involved in recruitment, career progression and establishing and evaluating research programmes are composed of at least 40 per cent of the under-represented sex. The 2014 ERA progress report indicates that the average share is currently 33 per cent. Gender imbalance in expert groups and in decision-making committees is thought to have further impact on the consistent and appropriate consideration of the gender balance in basic and applied research (German Federal Government, 2014). The proportion of organisations whose leaders are women is 18 per cent on average, with a high degree of variation among countries and where about half of Member States fall below the EU average. Member States are expected to increase the proportion of women at all career stages, and particularly in leadership positions and on executive boards of science organisations. Under Action 4b, RFOs are expected to work further to introduce gender related evaluation criteria for funding. Member States should look to remove institutional and cultural barriers that directly or indirectly prevent more gender-balanced decision making. This is expected to result directly in improved access to funding for female researchers and more gender-balanced evaluations. These activities may increase the share of female researchers across career stages and in research fields where women are particularly underrepresented (e.g. information sciences, engineering, and mathematics). Outcomes may also include improvements in the contractual situation of female researchers (e.g. the share of permanent versus non-permanent contracts compared to male researchers). An expected long-term impact of activities under this priority area is improvement in the labour market where there is a deficit in skilled labour and inefficient use of the qualified female labour force within the EU (EC, 2012b and 2013d) ERA priority 5 optimal access to and circulation and transfer of scientific knowledge The Commission has identified clear challenges and problem areas to be addressed under priority area 5 (EC, 2012): knowledge generated through research is not accessible throughout the research community due to institutional and infrastructural barriers; limited information is freely available to researchers in the public domain; the cost of accessing knowledge is high for smaller institutions, RPOs in less-advanced Member States and small and medium-sized enterprises (SMEs); knowledge transfer between academia and the private sector is unsatisfactory; and the lack of EU-wide digital infrastructure to manage the access to and maintenance of scientific knowledge is keeping costs for accessing knowledge high and specifically prevents institutions in less-advanced Member States from catching up. Priority 5 objectives include effective knowledge transfer, which is expected to contribute to open innovation and increase the speed of scientific discovery and knowledge spill-overs between academia and industry. Action 5a aims at improving open access to publications and access to data resulting from publicly funded research. Currently, only 44.6 per cent of the average share of research funders have strategies in place to support this (EC, 2014c). 2 April

21 ERA survey results in 2014 indicate that funding for open access to data is not a common practice in RFOs. Among those Member States whose funders support it, the average share of funding organisations frequently supporting open access activities is 28.1 per cent. Member States are therefore expected to provide legal frameworks for open access, which may result in RPOs making scientific research available in online repositories and subsequently a higher total number of scientific publications available through open access. Action 5b on open innovation and knowledge transfer between public and private sectors sets out objectives for Member States related to strengthening the connections between science and industry and on the role of public-sector research in open innovation. RFOs should systematically fund knowledge transfer activities as part of research projects and incentivise RPOs to support knowledge transfer through institutional support (e.g. through the introduction of technology transfer offices). These activities and short-term outputs are expected to result in further joint research developed between the private sector and RPOs as well as increased patenting and licensing revenues for RPOs. Greater spill-over effects to support the development of new products and services are expected long-term impacts under this action. Actions 5c and 5d seek to harmonise policies for public e-infrastructures and associated digital research services. RPOs improving access to public e-infrastructures using federated identities may over the long-term reduce the administrative costs of accessing scientific knowledge and computing power International dimension outside ERA The pace of scientific and technological change worldwide has increased pressure on the competitiveness of the European scientific community. Emerging economies continue to build research and innovation capacity. The share of the BRIC countries in global R&D expenditure doubled between 2000 and The European Commission has recognised these developments and warned that a lack of collaborative approaches with these countries poses significant challenges and that there are risks that the scientific community in Europe is falling behind (EC, 2012c). The cross-cutting priority on international dimensions outside the ERA is supported by a Commission Communication to enhance and focus EU international cooperation in research and innovation (EC, 2012). Member States have increasingly opened their research programmes to international cooperation, but fragmented national approaches to identifying and securing international talent may have resulted in reduced EU competitiveness and hampered access to foreign markets in technology-driven sectors (EC, 2012c). Under this cross-cutting priority area, Member States are encouraged to foster openness for international cooperation to maximise EU research potential. Most Member States already have national strategies for international cooperation on research and innovation (EC, 2014). But national level initiatives could be strengthened through greater coordination. The Commission supports a number of initiatives like the Strategic Forum for International Cooperation in Science and Technology (SFIC) in this regard (EC, 2012c and 2014e). RFOs are expected to increase the R&D budget going to third countries and work with Member States to develop collaborative programmes with third countries. Outcomes of these efforts may include an increase in research projects with third countries and improved international mobility of scientific knowledge and research results. Long-term impacts such as improved overall capacity to tackle global challenges and improved attractiveness of the EU as a location for researchers, companies and investments in R&D will benefit the EU economy and improve its attractiveness as an R&D location. The EU can also leverage the capacity international infrastructures and resources to tackle global challenges in R&D by strengthening the level of international cooperation in research. 2 April

22 3 Indicator identification and appraisal The next stage of the research involved a review of available indicators relevant to ERA monitoring in order to highlight data collection opportunities and map these across the priority areas. This also enabled an assessment of data requirements, where data best suited to ERA monitoring were insufficiently robust, complete or available. The research documented the information available for each indicator, the source, and frequency of collection. The indicators were then evaluated for suitability in relation to their relevance, reliability, availability, completeness and frequency of collection. The study team assessed the indicators against each criterion based on information provided in previous studies and expert reviews of ERA monitoring arrangements as well as the teams own judgment. The output is an inventory of indicators, organised by ERA priority, with an appraisal of their suitability for monitoring and evaluation, and a final suite of indicators proposed as a core set to measure progress across ERA priorities. There may also be indicators where data availability is currently unsatisfactory, but that would be valuable ERA indicators if information were available. The team considered such indicators and proposed that they could be included, where appropriate, if data collection were improved. The proposed indicators are intended to be illustrative of what the Commission could use in the future and provide a basis for discussion with Member States, RPOs and RFOs and other stakeholder organisations. 3.1 Approach of the appraisal For each priority area, the study team conducted an appraisal of available or potential indicators. The main data sources used for this exercise were Eurostat, the ERA Survey of RPOs and RFOs, the Innovation Union Scoreboard and Organisation for Economic Cooperation and Development (OECD) statistics. The appraisal also considered other known and potentially relevant and complete data sources in particular areas. Most notably, data from the second phase of the project on mobility patterns and career paths of researchers (MORE2) were considered for priority 3, and She Figures 10 for priority 4. Other sources that were considered included information captured by DG RTD and the Joint Research Centre (JRC) for the European Semester, as well as data captured by the Research and Innovation Observatory (RIO). The team conducted an appraisal of the indicators and data sources, identifying a smaller set of the best available indicators for use in ERA monitoring. The indicators were subdivided into input, output and outcome/impact indicators, with the aim of selecting one indicator for each category per ERA priority area, reflecting progress on as many of each area s actions as possible, resulting in a final selection of three indicators for each priority area, with one indicator respectively reflecting the best available input, output and outcome/impact indicators to capture progress on ERA priorities. In order to move from a large number of possible indicators to a final selection of three per priority, appraisal of each indicator involved a simple four-point scoring system on all key criteria that needed to be considered. Scores for the reliability and relevance of indicators carried a double weighting, given their exceptional importance. The scoring system is provided in Annex 8 along with the indicator maps and complete appraisal of all of the identified indicators. The final indicator chosen was chosen based on the overall score and taking into consideration potential limitations. This meant for instance that any indicator that received a 10 She Figures report on the situation of women in science and research through a set of indicators that assess the participation of women at all levels and in all scientific disciplines. Data collection is undertaken every three years, starting in 2003, by DG RTD in cooperation with the Helsinki Group and its sub-group of Statistical Correspondents (EC, 2012d). 2 April

23 low score on any one criterion would not be included in the final selection, even if it had the highest overall score. Further considerations are discussed where the appraisal revealed a choice to be made between two or more high-scoring indicators. Indicators that were fundamentally relevant but currently not available or where data quality was poor are discussed separately. The indicators selected (three per priority area) represent what the study team believes should be the minimum number and range of indicators and as such the simplest possible reference tool that can currently be derived from the wide range of available indicators. As explained below, a consideration here was also to ensure that the smallest possible number of indicators could reflect the fullest possible range of actions under each priority. As such, it was critical to identify indicators that address several actions where possible, meaning that indicators should not be read as representing linear progression across each action (this would involve an absolute minimum of three indicators for each of the 19 actions, or 57 separate indicators). The breadth of each ERA priority, as well as the variety of actions contained within them, could be represented by a larger number of indicators. Additional potential indicators are noted where relevant, whilst areas not covered by currently available indicators are also identified. The resulting indicator matrix is therefore a substantive tool for immediate consideration and use, but should not be viewed in isolation from the whole of the analysis presented here Indicator maps and links between actions and indicators The study team developed an indicator map for each ERA priority area, which plots indicators across inputs, outputs and outcomes to identify where there are data and indicators available that are or could be used to measure progress in each priority area. The indicator maps are presented in Annex 8. The indicators considered included: Indicators identified by DG RTD for particular consideration. Indicators which have been identified in reports prepared by expert groups reviewing the ERA monitoring mechanisms (e.g. JRC, 2013; EC, 2009a; Haegeman et al., 2012; Doussineau et al., 2013). Indicators available from existing datasets (e.g. OECD, Eurostat, the Innovation Union Scoreboard, etc.) Indicators that can be derived easily from the activities identified through the intervention logics. These indicators could be useful to fill gaps in the monitoring framework, particularly in the measurement of outputs and outcomes. Indicators that have been used in ERA progress reports are presented in the indicator maps. They are not appraised here: instead, the results of the analysis in this section presents possible alternatives and extensions to the indicators already in use. Indicators that have been used in the ERA survey for RPOs are also presented. Assessment of these indicators is out of scope for this study, but are identified in the indicator maps to demonstrate where information about research performing organisations is being used to assess ERA progress. Actions under each ERA priority are included in the indicator selection. Indicators should not be read directly against actions, however, and are not intended to suggest a link between inputs, outputs, and outcomes/impacts. Moreover, outcome/impact indicators consistently overlap across the different ERA priorities. Linking individual inputs to specific, long-term impacts is problematic for almost all priority areas. Similar effects can occur for outputs, and even for inputs there is not always a clear link between each indicator and a single action. Most often, this is due to the broad nature of the actions, as well as to the multitude of identified indicators, which are often highly relevant to a priority area, but do not always neatly address one action specifically. 2 April

24 Instead, the indicators selected are intended to highlight the availability of both broad and more focused indicators relating to each priority area. In practice, this means that an indicator s ability to highlight progress on multiple actions, or the ability of a selection of indicators to cut across as many actions as possible, are critical criteria to make the best use of available data to monitor the progress of the ERA as a whole The potential for composite indicators As an alternative to selecting one input, output, outcome and/or impact indicator for each of the ERA priority objectives, it is possible to construct a composite indicator. A composite indicator summarises information for pre-selected variables. The variables of choice can include input, output, outcome, and impact variables or can be limited to a selection of outcome or other variables only, for example. It also is possible to construct a composite indicator that captures the different dimensions of the ERA priorities. Synthesizing information in this way can facilitate tracking a country s overall score on ERA priorities over time relative to another country. As a result, composite indicators can be powerful tools for communication. Despite the advantages of composite indicators there are also a number of pitfalls. In order to construct composite indicators for each of the ERA priority areas that are comparable over time and comparative across countries there are a number of prerequisites. First, the underlying data must be available for all countries for all years under consideration. 11 Different methodologies can be used to construct composite indicators, including factor analysis (which also has a multitude of options), simple aggregation to the mean and correlation analysis. The appropriate methodology is dependent on the theoretical framework and the data characteristics. If the data characteristics are substantially different across the different ERA areas and it is the intention to construct different sets of composite indicators, different methodologies may be recommended across the ERA areas. The study team does not recommend constructing composite indicators at this stage, however, because they are less powerful when it comes to representing specific policy actions and recommendations on ERA priority areas. Measuring progress or lack thereof over time usually requires analysing changes in single variables that capture different aspects of a given ERA priority area. For example, the input variables on gender equality includes a variable measuring the proportion of funding allocated for projects that integrate gender aspects to science and technology research and a variable that captures the number of applicants and beneficiaries of research funding (by sex). Countries may achieve different levels of progress on each of these ERA aspects. When working with composite indicators it is more challenging to interpret the progress made on different ERA priority areas and to recommend a specific policy action. 3.2 Priority 1 - indicator assessment ERA priority 1 focuses on effective national research systems. Thirteen indicators (excluding indicators used in previous ERA progress reports) were identified for this area (see Annex 8, section A8.2). The ERA actions relate closely to funding issues, therefore the suggested indicators are focused on finance. Useful, reliable and robust data covering ERA Member States are available through Eurostat and the OECD (Science and Technology Indicators and Science and Technology Outlook). These databases cover all Member States in most cases and offer time series data. A disadvantage of some of the Eurostat and OECD indicators is their generic focus, which makes them helpful as headline figures but not always fully relevant to specific actions or ERA priorities. Indicators based on the two-part ERA Survey of, respectively, Research Funding and Research Performing Organisations 11 In the case of missing data, it is possible to use imputation but additional care has to be taken to interpret the resulting composite indicator. See also OECD (2008) Handbook on constructing composite indicators: methodology and user guide for a general guideline on constructing composite indicators. 2 April

25 (hereafter ERA Survey) might be better at measuring progress for specific ERA priorities, but the robustness of such indicators is a drawback: on some questions, response rates have been high for previous studies and there are complete data for all or almost all Member States, but this is not always the case. Overall, response rates have been low in some countries, which is a problem that needs to be addressed in order to obtain more robust data in the future. Total government budget appropriations or outlays for R&D (GBAORD) as a per cent of GDP could be an appropriate input indicator. Data are collected regularly by OECD and Eurostat and are highly reliable and revised annually. Whilst it is a general indicator that leaves considerable space for interpretation, it is by far the most robust of the available selection, and provides an indication of the resources invested in the research system. The share of the national GBOARD allocated through project-based funding (as opposed to institutional funding) is the preferred choice for an output indicator, as these data are collected through the ERA survey and has received high response rates. This indicator could be considered to be an input rather than output of the research system. But the decision to make a proportionately larger amount of funding available on a competitive basis suggests that researchers are deemed able to compete for such funding, and can do this to the extent that institutional stability is possible, making large amounts of block funding no longer essential. Immediate outputs for priority 1 yielded only a small number of possible indicators. Nevertheless, the ability of researchers to secure funding and the presence of increased competitive funding signify improvements in the research system. Combined with reliable and robust indicators to measure these aspects, the share of national GBOARD is a reasonable option. There are several potential outcome/impact indicator options: trademarks as a per cent of GDP; number of patents per total public R&D expenditure; and revealed technological advantage in selected fields (e.g. bio- and nanotechnology, ICT and environment). The last indicator is a closed measure of the impact of actions taken in R&D. But establishing a causal link between the input and impact is difficult. Number of patents per total public R&D expenditure was selected as the preferred indicator for measuring efficiency of public spending for innovation output. The final selection of indicators for ERA priority 1 is summarised in Table April

26 Table 3.1 Priority 1 proposed indicators Type Indicator Data source Comments/ explanation Input Output Total GBAORD as per cent of GDP Share of national GBOARD allocated through projectbased funding (as opposed to institutional funding) OECD Main Science and Technology Indicators, Eurostat Good overall indicator of the resources allocated to research, taking into consideration the different sizes of individual MS. Available annually, time series, one-year or two-year time lag. ERA Survey Project-based funding is an accepted means by which to generate competition among researchers and improve overall quality and productivity. Collected in the ERA Survey 2014; inclusion in all future ERA Surveys likely. Outcome/ Impact Number of patents per total public R&D expenditure Eurostat Measures efficiency of public spending for innovation output. Good measure of effectiveness. Available since 2000 (available as a single indicator until 2009, both components still available separately), regularly updated Data collection needs and opportunities The analysis has not highlighted any immediate data collection needs for priority 1, although the selected output indicator highlights that continuation of the ERA survey in some (simplified) form is important to ensure the availability of indicators. At present there are suitable input, output and outcome/impact indicators, which are relevant to all specific actions. All of the indicators identified as relevant to the priority objectives are robust. Two indicators assessed relate to research evaluations: share of national institutional funding allocated based on institutional assessments; and share of block and institutional funding allocated using performance-based criteria, as share of national GBAORD. A country s use of performance-based research funding systems could be a useful subject theme for an indicator to assess ERA priority 1. But there are many limitations that need to be considered, including: Not all research evaluations are institutional: some countries (such as the United Kingdom (UK)) distribute a large share of institutional funding through evaluations, but evaluation is conducted at the level of the discipline, not the institution. Some countries (especially Nordic countries) have sophisticated and regular evaluations of both fields and institutions, but do not base much if any of their funding decisions on these evaluations. Nevertheless, the evaluations are useful in terms of quality control and identification of research strengths and priorities. An indicator should ideally capture such endeavours, or at least not make them appear insignificant. Whilst there is broad consensus that some degree of performance based institutional funding is helpful in terms of achieving better and more targeted outcomes in the research system, there is little evidence to suggest that the extent of positive outcomes is dependent on the amount or share of funding distributed in this way: some degree of non-performance based funding can be critical to ensuring long-term stability of institutions. This is especially important for institutions wishing to plan long-term research endeavours, as well as institutions with substantial teaching duties. 2 April

27 Ultimately, there is a potential need to design an indicator capable of capturing information on research evaluation and assessment-driven funding allocation. But the complexity of the subject matter, as well as the diversity of both institutions and assessment and evaluation mechanisms make this a formidable task. In the ideal case, qualitative investigation into individual Member State research assessment models may be preferable to a numerical indicator, but notwithstanding this, currently available indicators do not offer a suitably robust compromise. 3.3 Priority 2 - indicator assessment Twenty indicators were identified and appraised for ERA priority 2, Optimal transnational cooperation, competition and research infrastructures (see Annex 8, section A8.3). The two main data sources are the OECD Science, Technology and Industry Scoreboard and the ERA Survey. Member State coverage and data reliability are advantages of the OECD databases. Possible indicators based on the ERA Survey could provide more specific information related directly to intra-eu activities aimed at greater integration and cooperation. The ERA Survey is an especially valuable source for inputs indicators. The degree to which Member States participate in the European Strategy Forum on Research Infrastructure, expressed in the survey as participation in at least one of the research infrastructures (RIs) identified by ESFRI, provides useful insight into this particular aspect of ERA. The best available output indicator for this priority is the share of public funding allocated to transnational R&D cooperation, which is collected by Eurostat. The true figure of funding used in transnational R&D may well be much higher than what is reported, however, since many research projects may have transnational dimensions even though the funding was not explicitly allocated with such a condition attached. But identifying the full amount of R&D funding used in transnational endeavours requires information from funders that is potentially confidential. Given this difficulty, it is preferable to opt instead for the share of funding explicitly allocated for transnational work, as this is a readily available figure, and reflects not only the extent of transnational activity, but also highlights the efforts made (in terms of resource provision) to create more transnational cooperation. The OECD commissions several useful outcome/impact indicators that are reliable, cover most Member States and provide time series data. Their design does not focus specifically on monitoring ERA-related priorities but the overall effects of international cooperation overlap well with the desired results of ERA priority 2, so OECD data can be used here, particularly through the following indicators: impact of internationally mobile scientists, inflows versus outflows; international collaboration in science and innovation; cross-border ownership of patents; technology balance of payments as a per cent of GDP; and international technology flows of royalties and licence fees. Cross-border ownership of patents has been chosen as the preferred indicator. It captures the degree to which Member States collaborate internationally in developing technology and innovation. The OECD Science, Technology and Industry Scoreboard is a basic reference, but the OECD statistics and European Patent Office (EPO) statistics are more suitable alternative sources for exploring this type of indicator. International collaboration in science and innovation is another viable measure, though a more specific aspect, such as patent ownership relates better to private as opposed to public R&D. The alternative would give a more general picture that would also be influenced by strictly blue-sky research. The preferred indicator between these two could not be identified in this analysis. 2 April

28 With respect to research infrastructures, available sources do not offer enough data to assess impact on the economy and society. The ESFRI proposal referenced Indicators of pan-european relevance to research infrastructures and the ESFRI expert group on indicators developed a toolkit for the evaluation of the pan-european relevance of ESFRI roadmap projects and future candidate entries (Rossi, 2013). Some of the indicators are newly constructed and their usefulness depends on data availability, which often requires new data collection within research organisations. When their availability and the frequency of data collection are established, these could be considered for future ERA monitoring activities. Table 3.2 Priority 2 proposed indicators Type Indicator Data source Comments/ explanation Input Per cent of MS participating in the development of at least one of the RIs identified by ESFRI Output Share of public funding Eurostat allocated to transnational R&D cooperation ERA Survey Useful indicators because the ERA Survey is tailored to assess ERA progress. Included in ERA Survey; inclusion in future ERA Surveys likely. This indicator provides a measure both of the resources explicitly allocated for transnational cooperation, and gives an indication of the minimum degree of transnational R&D (the real figure being potentially higher). Outcome Cross-border ownership of patents OECD Science, Technology and Industry Scoreboard Available through Eurostat. Data collected since 2007, latest data from Useful indicator for this priority because it captures international innovation aspects. Available annually. Priority 2 does however present a special case, in that its breadth invites a wider selection of indicators. It has furthermore been split into two sub-priorities, relating respectively to transnational cooperation and ESFRI. Whilst the above table presents the final selection of indicators, the inventory of possible indicators may also be used to cover the two subpriorities separately, as presented in Table April

29 Table 3.3 Priority 2 proposed indicators along sub-categories Type Indicator Data source Comments/ explanation Input ESFRI: Per cent of MS participating in the development of at least one of the RIs identified by ESFRI ERA Survey Useful indicator because the ERA Survey is tailored to assess ERA progress. Included in ERA Survey; inclusion in future ERA Surveys likely. Output Transnational cooperation: Degree to which MS engage in transnational cooperation via an EU framework programme ESFRI: Per cent of research performers experiencing problems accessing RIs ERA Survey ERA Survey Highlights the extent to which the framework programme is drawn upon to achieve greater transnational cooperation Included in ERA Survey; inclusion in future ERA Surveys likely. Highlights the extent to which access to research infrastructures is facilitated in individual MS. Note: on this indicator, lower values will indicate preferable scores. Included in ERA Survey; inclusion in future ERA Surveys likely. Outcome Transnational cooperation: Share of public funding allocated to transnational R&D cooperation ESFRI: Number of implementation phase ESFRI projects in which each MS is a partner Eurostat ERA Expert group (Data available through ESFRI) This indicator provides a measure both of the resources explicitly allocated for transnational cooperation, but also gives an indication of the minimum degree of transnational R&D (the real figure being potentially higher) Available through Eurostat. Data collected since 2007, latest data from Highlights overall success in MS participation in ESFRI. Included in ERA Survey; inclusion in future ERA Surveys likely. Transnational cooperation: Cross-border ownership of patents OECD Science, Technology and Industry Scoreboard Useful indicator for this priority because it captures international innovation aspects. Available annually Data collection needs and opportunities The indicator appraisal for this priority area has yielded several indicators that could be used immediately for monitoring, but the analysis has also highlighted some data collection needs, due to the fact that some potentially useful indicators have been suggested by the ERA expert group, but the necessary data are not collected systematically. Share of national public funding for R&D transnationally co-ordinated, expressed as a percentage of the GBOARD. The share of national funding for which the condition of transnational coordination applies is likely to be small, and subject to many external factors. But the involvement of non-national research agencies / partners in the framing of national research priorities provides a useful indication of the extent to which a country is outward-looking and 2 April

30 receptive to international perspectives and priorities. Depending on the exact definition, measurements related to this theme could either contribute to input or output indicators. Involvement of non-national research agencies represents an area for further discussion. Share of the national GBAORD invested in the construction and operation of research infrastructures listed on the ESFRI roadmap. This is a good example of measuring EU added value and a very relevant input indicator. The EU has a long history of developing international research infrastructure, and with the increasing importance of large and costly facilities in the majority of research fields this tradition is likely to intensify. Relevant data have been collected previously through the ERA Survey, but only three Member States provided estimates. Further efforts would be needed to gather more complete data. Amount and share of joint research agenda initiatives that address grand challenges and are subject to common ex post evaluation. This indicator would highlight a key aspect of collaboration, whilst also demonstrating the level of importance of research activities in relation to current social and political issues. But data are not currently collected for this indicator. A robust definition of relevance to grand challenges must be agreed to avoid different interpretations. Share of national GBAORD allocated to transnationally coordinated research based on grand challenges. This could be a valuable indicator related to Member States overall expenditure. A common definition of relevance to grand challenges must be agreed. 3.4 Priority 3 - indicator assessment This ERA priority is particularly challenging to monitor because the concept of an open labour market is multi-faceted, encompassing national and transnational movement between research institutions, between research and industry, and several other dimensions contained for instance in the European Code of Conduct for the Recruitment of Researchers. The five distinct actions under this priority illustrate its breadth. Seventeen indicators were identified and appraised for this priority area (see Annex 8, section A8.4). The percentage of research funding available for mobility scholarships and stipends of total research funding is the preferred input indicator, as it reflects resources invested to achieve this priority and is reliably collected through the MORE2 survey. There are two suitable choices for output indicators, which address fundamentally distinct issues. The number of researcher posts advertised through EURAXESS by Member State could serve as a useful indicator. There are currently some limitations to this, which are discussed below. The share of research organisations that have been acknowledged by the Commission s Human Resources (HR) Excellence in Research award. This indicator would especially reflect institutional awareness and openness. There are several potential outcome/impact indicators, although each one focuses on a different but important aspect of an open labour market. Time spent abroad and outside of academic institutions could be combined as a composite indicator. But the most robust figures are collected by different organisations that use different data collection approaches. The degree of international mobility of researchers is a good indicator of an open labour market. As such, the proportion of doctoral candidates with citizenship from another EU Member State has been chosen as the preferred outcome/impact indicator for priority 3. The indicator shows researcher mobility within the EU at an early career stage. As such, it is a proxy for attractiveness of individual Member State research systems, the presence of 2 April

31 mechanisms capable of attracting non-national EU researchers, and the extent of mobility at this career stage. Data to construct this indicator are available, and it has been suggested for inclusion in the IU scoreboard and is supported by DG RTD. Eurostat s data on researchers working in the business sector is a plausible alternative. Both options are considered in Table 3.4. Table 3.4 Priority 3 proposed indicators Type Indicator Data source Comments/ explanation Input Per cent of funding MORE2, available for mobility JRC scholarships and stipends out of the total funding for research Output Share of organisations with EC HR Excellence in Research Acknowledgement Outcome/ Impact Proportion of doctoral candidates with a citizenship of another EU MS EC Website, JRC IU Scoreboard (pending) The indicator gives an overview of the relative degree of spending for mobility, but it might be difficult to identify the sources and collect the data. Updated annually. This indicator highlights acknowledgement of efforts to ensure progress related to several actions of priority 3 Updated with high regularity. The share of non-national doctoral candidates as a percentage of all doctoral candidates serves as a useful indicator of the openness and attractiveness of a research system. Updated annually or every two years. Alternative: Researchers working in the business sector Eurostat This alternative indicator highlights outcomes at a different and important level of mobility and open labour markets Updated regularly Data collection needs and opportunities The ERA expert group has identified the share of job offers within the national public research system published on EURAXESS or equivalent websites as a potential output indicator. This was noted above as a highly relevant indicator, but there are some concerns with the indicator which would need to be addressed: Whilst numbers of jobs advertised through EURAXESS are recorded for individual countries, there is a serious problem of adjustment. The Researchers' Report for EURAXESS Jobs (2014) has constructed an indicator adjusting for size of Members States research systems: Researcher posts advertised through the EURAXESS Jobs portal per thousand researchers in the public sector. But whilst this provides one critical level of adjustment, it does not reflect different recruitment patterns and frequencies in different countries: for example, systems that typically have more short-term contracts and fewer tenured positions may score higher. In order to adjust for this, it would be necessary to express the number of EURAXESS postings not per thousand researchers, but as a share of overall appointments to new posts. Additionally, some Member States might have their own commonly used portals for advertising research posts (e.g. the UK s whilst other might not. The scale of this issue is unclear at this point, but it may lead to problematic results. Although advertising job posts is an important aspect of an open labour market, open publication of appointments is not equivalent to appointments being decided based on merit. 2 April

32 An additional limitation for this indicator is that it only relates to the public sector, whilst the ERA objectives are multi-sectoral. Nevertheless, EURAXESS is a valuable resource that could be drawn on in future to better inform progress on ERA priority 3 if used in concert with other metrics. An additional possible data collection need relates to the importance of joint research for mobility, which is noted throughout the literature, and begins as early as doctoral training (EC, 2011b). As such, the share of joint projects or publications encompasses opportunities for greater mobility and openness in several different inter-institutional contexts. Vertesy and Tarantola (2012) suggest the possibility for targeted indicators on co-publication, to be drawn from data sources such as Scopus. This would be a major data collection project, but could ultimately contribute to a better understanding of where cross-institutional co-publication occurs. 3.5 Priority 4 - indicator assessment The objective of priority 4 is to achieve greater gender equality (in terms of socio-economic circumstances and equal opportunity) and gender mainstreaming. Specific objectives include creating a supportive legal and policy environment, removing legal and other barriers to the recruitment, retention and career progression of female researchers and addressing gender imbalances, and ensuring that at least 40 per cent of participants in recruitment/career progression decisions are women. The objective of gender mainstreaming is to strengthen the gender dimension in research programmes. The study team appraised 20 indicators in this priority area (see Annex 8, section A8.6). There are disproportionately more indicators related to gender balance than to mainstreaming primarily because gender mainstreaming is a more qualitative concept, referring to the content and scope of research, as well as to gender representation in managerial and evaluator positions, whose status and functions differ significantly between Member States, unlike more standard academic roles. As such, indicators on gender balance are significantly more robust, whilst numerical measurement of gender mainstreaming requires further deliberation. The She Figures reports published by the European Commission provide a valuable source of gender statistics in the area of research and innovation. She Figures build on several data sources including Eurostat, Statistics on Income and Living Conditions (SILC), and International Labour Organisation (ILO) data. The first choice of input indicator under ERA priority 4 is the proportion of female PhD graduates according to the International Standard Classification of Education (ISCED 6). This indicator reflects the degree to which there is a gender-balanced pool of job market candidates, and whether gender balance is something that Member States are able to facilitate from the point of early career development onwards. The indicator is revised annually and is available for all Member States. An alternative indicator is the number of applicants and beneficiaries of research funding by gender. There are several output indicators that can be considered, which reflect gender balance in the academic workforce: proportion of female academic staff; share of female researchers on temporary contracts vs. non-temporary contracts across career paths; distribution of researchers in the higher education sector, by sex and age group; and distribution of researchers across sector, by sex. The output indicator selected by the study team is the proportion of female academic staff. The indicator is based on data that all Member States are able to provide and gives an overall snapshot of the extent to which gender balance is achieved across academic grades. 2 April

33 Whilst promotion of gender equality and greater numbers of female PhD students might result in the immediate output of better gender balance across the sector, these data may hide certain forms of discrimination, for example, gender imbalances amongst senior staff. As a result, the proportion of women in grade A (professor) positions has been chosen as the preferred outcome/impact indicator. The final selection of indicators for ERA priority area 4 are summarised in Table 3.5. The data quality for the stated sources is still improving, but close to being comprehensive. Furthermore, their focus is on the HEI sector. Nevertheless, the chosen indicators represent the most robust and relevant selection of the available options. Table 3.5 Priority 4 proposed indicators Type Indicator Data source Comments/ explanation Input Proportion of female PhD (ISCED 6) graduates She Figures (2013) based on Eurostat - Education Statistics (online data code: educ_grad5); Italian Ministry of Education (IT - MIUR) ( ) The indicator reflects the degree to which there is a gender imbalance in the distribution of PhD graduates. The indicator should be used in concert with other indicators to give a complete picture of ERA priorities (e.g. actual employment of female researchers after graduation). The data is available for all MS. Output Proportion of female academic staff Outcome/ Impact Proportion of women in grade A (professor) positions. Annual data She Figures, based This indicator reflects gender balance in the on data from WiS academic workforce. Data are missing for some database (DG MS. RTD) Data collected every three years She Figures Degree of gender balance in the distribution of researchers in the workforce. Data available for most MS. Data collected every three years Data collection needs and opportunities Data collection initiatives such as She Figures provide robust indicators for this priority area. But the majority of available indicators relate predominantly to gender balance. In this area there are no further data collection needs. But while gender balance can be reported through the use of numerical indicators, it is more challenging to identify indicators that reflect gender mainstreaming, which refers to the content and scope of research and research programmes, as well as the gender balance of those involved in their design. One suggested indicator is the proportion of funding allocated for projects that integrate gender aspects into science and technology research (also known as gender mainstreaming the science / gender dimension in research content). This indicator highlights gender awareness amongst researchers. But data are not collected for this indicator. There are additional challenges with using this indicator such as how to define the integration of gender aspects into research. There might be several different forms this might take, and it might occur to different extents that could not be robustly quantified. There is also likely to be considerable variation in the opportunities to include the gender dimension across different research fields. 3.6 Priority 5 - indicator assessment Fifteen indicators were appraised for ERA priority area 5 (see Annex 8, section A8.6). Four of these indicators do not have any available datasets; these are discussed in section April

34 The preferred input indicator is the share of funders funding open access to publications. Data for this indicator are collected through the ERA survey and provide insight into how open access is being pursued and the extent to which funders are putting resources in place to ensure greater availability of research to wider stakeholders. As open access publication is not yet as systematised as research through traditional publication channels, output and wider impact indicators on, for example, availability or use of open access publications are inherently problematic. The appraisal identified R&D in HEIs / PROs funded by business as the preferred output indicator. This indicator is a monetary measure of business interests according to the R&D they perform and fund in the higher education sector (HERD) and the government sector (GOVERD). The data for this indicator are robust and available as a time series through Eurostat. But this indicator does not make a distinction between large and small firms, and the capacity of large firms to fund R&D in HEIs and PROs is much greater than that of small firms. Additionally, the data do not distinguish between the degree of cooperation between firms funding R&D in HEIs / PROs. Seven outcome / impact indicators were identified for this priority area, two of which could be used in a future monitoring framework. These indicators are firms co-operating with HEIs and firms co-operating with PROs. They represent reasonable proxies for private firms propensities to work with HEIs and PROs, respectively. But while these are useful indicators, the data do not distinguish between large and small firms, and large firms are more likely to cooperate with HEIs and PROs than small firms due to their R&D capacities. Available data do not enable a distinction between the level and extent of cooperation (Finne et al., 2011). The preferred outcome / impact indicator in this priority area is the number of public-private co-publications per million of the population, which is a proxy for public-private research linkages and active collaboration activities between business sector researchers and public sector researchers. This indicator combines data from two sources: the Centre for Science and Technology Studies (CWTS), Leiden University (Thomson Reuter) and Eurostat. This indicator is particularly useful for looking at outcomes / impacts because it addresses two of the main challenges of this priority area. The first of these is effective knowledge transfer, which is expected to contribute towards open innovation and the second, open knowledge transfer between the public and private sector. There are two issues with this indicator, however. First, the definition of private sector used in this context does not include private medicine or the health sector and second, publications are assigned to the country / countries in which the business or other private sector organisations are located and not where the public sector organisation / institution is located (OECD, 2014b). Table 3.6 Priority 5 proposed indicators Type Indicator Data source Comments/ explanation Input Share of funders funding open access to publications Output R&D in HEIs / PROs funded by business Outcome/ Impact Public / private copublication per million of the population ERA Survey Open access publication is an essential component of ensuring circulation, access to and transfer of knowledge. Eurostat CWTS (Thomson Reuter) and Eurostat Included in ERA Survey; future inclusion highly likely. Robust and reliable data source. Provides an indication of the extent of public-private collaboration. Latest data Data collection methods are reliable and robust. Indicator addresses several challenges highlighted in this priority area, and represents public-private collaborations. Latest data Regular updates through Eurostat. 2 April

35 As is the case with priority 2, priority 5 is also especially broad and has been divided into two sub-priorities: open access and knowledge transfer. However, in this case, the data collection needs are so extensive that it is impossible to propose a robust and complete alternative matrix of indicators to deal with both sub-priorities separately. The problem arises in relation to open access specifically. Data collection needs in this area are discussed below. Table 3.7 presents an alternative matrix that fully addresses the sub-priority on knowledge transfer, but shows where data collection needs for open access are currently required. Table 3.7 Priority 5 proposed indicators along sub-categories Type Indicator Data source Comments/ explanation Input Open Access: ERA Survey Share of funders funding open access to publications Open access publication is an essential component of ensuring circulation, access to and transfer of knowledge. Included in ERA Survey; future inclusion highly likely. Knowledge transfer: Share of organisations that has or uses a structure for knowledge transfer activities ERA Survey Indicates the resources available to engage in KT activities. Included in ERA Survey; future inclusion highly likely. Output Open Access: Data collection needed n/a n/a Outcome/ Impact Knowledge transfer: R&D in HEIs / PROs funded by business Open Access: Data collection n/a needed Eurostat Robust and reliable data source. Provides an indication of the extent of public-private collaboration. Latest data n/a Knowledge transfer: Public / private copublication per million of the population CWTS (Thomson Reuter) and Eurostat Data collection methods are reliable and robust. Indicator addresses several challenges highlighted in this priority area, and represents public-private collaborations. Latest data Regular updates through Eurostat Data collection needs and opportunities Whilst there are suitable indicators available to measure progress on ERA priority 5, there are also data collection needs for some actions. First, there is a gap with regard to indicators related to the ERA actions under this priority area. Priority 5 is composed of four actions: open access to publications, open innovation and knowledge transfer between the public and private sector, harmonising policies for public e-infrastructures, and uptake of federated electronic identities. The available indicators broadly cover the first three actions. But there are currently no indicators which address uptake of federated electronic identities. The topic of digital unique researcher identification, as well as a federated system common to all Member States is likely to become more 2 April

36 important, as proper attribution of work to researchers is essential for research assessment exercises. There is a need to consider how progress on this might be measured and who is best suited to undertake the necessary data collection. There are also indicators that could be used in this priority area, but for which data are not currently collected. This primarily relates to indicators on open access publication: Number and share of national research performing organisations with mandatory policies for open access to and preservation of scientific information (EC, 2013c). This indicator would provide a good measure of the first action under priority 5. The number and share of institutions with mandatory open access policies could be provided by Member States. A clear definition of the form and substance of these policies would be required. Share of open access publications compared to total output of Member States (EC, 2013c). National and international open access repositories and national statistics on scientific publications could be combined and drawn on to produce this indicator. Since open access publication is still a relatively new activity, there is a question as to whether this would capture all open access journals. Open access repositories are updated on a regular basis and often provide statistical information. National statistics on scientific output are available in most Member States although there is some variation, so this indicator may be restricted to Member States that publish national statistics on scientific publications. A further point worth highlighting on the issue of open access publication is the difference between gold and green open access, which respectively refers to whether or not researchers are charged money for their work to be published. In the gold standard model these costs can be prohibitive, and effectively shift the access cost from user to producer of the research. Archambault et al (2014) conducted a study touching on this subject, which could provide a starting point for regular data collection in the future. There are two further indicators that warrant consideration: Number and share of research performing organisations with interoperable and federated repositories (EC, 2013c). This would be a useful indicator to capture progress on action 3. Once again, this should be an un-problematic metric for Member States to provide, although a clear definition of an interoperable and federated repository would need to be agreed upon. This could be done in the form of a shortlist of specific systems in use. Share of research and development budget financed by the private sector. This indicator was suggested by DG RTD for consideration in this study, and would help to show the extent of transfer between the research and private sectors. Data for this can be provided by the ERA Survey. But it is difficult to express private sector investment as a share of the overall R&D budget: reduced government investment would artificially increase this indicator and, conversely, additional government investment in R&D would decrease it. An alternative way of looking at private sector R&D financing would need to be considered to address this problem. 3.7 International Dimension Outside ERA (Priority 6) - indicator assessment Eight indicators were assessed for this cross-cutting priority area on international dimensions outside ERA (see Annex 8, section A8.7). The ERA survey collects data on the share of public R&D allocated to collaborative programmes with third countries. This input indicator is recommended because it provides a clear idea of resources invested in international collaboration. 2 April

37 This priority area contains only one action, but it includes both international research projects (the products of research), as well as researchers themselves. Non-EU doctorate holders as a percentage of total doctorate holders is therefore a useful indicator of the research system s international status. More recent data need to be collected on this indicator because only 2006 and 2009 data are available. There are three options for outcome/impact indicator. Licence and patent revenues from abroad as a per cent of GDP is a proxy for the main challenge in this area, and is the preferred indicator: assessing international activity and cooperation between countries. Licences and patents represent the transfer of techniques in the context of the trade in technology, which is calculated from export data in which international transactions in royalties and licence fees can be separated out (Guellac and Potterie, 2001). Data are available from Eurostat as recently as The other two possible indicators are international co-publications per million of the population and the per cent of patents with foreign co-inventors. Both are proxies for international co-operation and research. But neither indicator identifies the nationalities of the first and second authors and so it is not possible to distinguish which countries have led each collaboration and the relative contributions of each nationality within the collaboration. The three selected indicators together capture research projects, researcher mobility, as well as returns from international cooperation and international demand. Table 3.8 International dimension outside the ERA (Priority 6) proposed indicators Type Indicator Data source Comments/ explanation Input Share of the public R&D budget allocated to collaborative programmes with third countries Output Non-EU doctorate Eurostat (via holders as a per the IU cent of total Scoreboard) doctorate holders Outcome/ Impact Licence and patent revenues from abroad as a per cent of GDP ERA Survey Collected through the ERA survey; useful because it excludes collaboration within the EU. Eurostat (via IU Scoreboard) Data collection needs and opportunities Included in ERA Survey. Only indicator in this category, data sourced from Eurostat so data is likely to be reliable and collection methods relatively robust. Data available for 2006 and More recent data collection would improve this indicator. Highest scoring indicator in this category. Updated annually or every two years, latest data This ERA priority area has a small set of robust indicators. There is one further indicator that may be considered, however. The proportion of researchers employed in each Member State that originates from non-european countries would be an indicator of the internationalisation of European research systems. Although many Member States are likely to hold this information, it is currently not being collected comprehensively. The indicator would require a precise definition to avoid variance in the data reported, for example, regarding whether PhD students should be included in this figure. 3.8 Indicator selection - summary The identification and appraisal of existing and potential indicators to monitor ERA progress yielded many possibilities. But there are disparities amongst them in terms of reliability, relevance, frequency of data collection, accessibility and completeness. As a result, the study team undertook an assessment of each indicator s strengths and weaknesses, in order 2 April

38 to select those best suited for ERA monitoring. Furthermore, given the breadth of each ERA priority area and the actions within them, indicator choices were calibrated to reflect progress in relation to all actions, where possible Selected input, output and outcome/impact indicators by priority area Table 3.9 presents the final selection of input, output and outcome/impact indicators for each of the ERA priorities. Each indicator is collected and updated with some degree of regularity and is available from accessible and reliable sources. As such, the indicators selected represent an indicator suite that can be used by a range of stakeholders to obtain a basic overview of progress across priorities. Table 3.9 Selected input, output and outcome/impact indicators by priority area Priority Input Indicator Output Indicator Priority 1: more effective national research systems Total GBAORD as a per cent of GDP (OECD) Share of national GBOARD allocated through project-based funding (as opposed to institutional funding) (ERA Survey) Priority 2: optimal Per cent of MS Share of public funding transnational participating in the allocated to cooperation and development of at least transnational R&D competition one of the RIs identified by cooperation (Eurostat) ESFRI (European Research Area and Innovation Committee (ERAC)) Priority 3: Open Labour Market for Researchers Priority 4: Gender equality and gender mainstreaming in research Priority 5: Optimal access to and circulation and transfer of scientific knowledge International dimension outside ERA (Priority 6) Per cent of funding available for research mobility scholarships and stipends of the total funding for research (MORE2, JRC) Proportion of female PhD (ISCED 6) graduates (She Figures, based on Eurostat) Share of funders funding open access to publications (ERA Survey) Share of the public R&D budget allocated to collaborative programmes with third countries (ERA Survey) Share of research organisations with EC HR Excellence in Research Acknowledgement (EC web site, JRC) Proportion of female academic staff (She Figures, based on data from WiS database) R&D in HEIs / PROs funded by business (Eurostat) Non-EU doctorate holders as a per cent of total doctorate holders (Eurostat via the IU Scoreboard) Outcome/ Indicator Impact Number of patents per total public R&D expenditure (Eurostat) Cross-border ownership of patents (OECD) Proportion of doctoral candidates with a citizenship of another EU MS (IU Scoreboard, pending) Alternative: Researchers working in the business sector (Eurostat) Proportion of women in grade A (professor) positions (She Figures) Public / private copublication per million of the population (CWTS and Eurostat) Licence and patent revenues from abroad as a per cent of GDP (Eurostat via the IU Scoreboard) This indicator suite is capable of highlighting whether for each ERA priority there is evidence of resources being made available to achieve it (input), immediate observable results of such efforts (outputs), and wider improvements that match the fundamental goals of each priority 2 April

39 (outcomes/ impacts). The following points should be considered in the interpretation and usage of the indicator suite. Across inputs, outputs and outcomes / impacts there is no direct linear progression. Doing so would involve having separate sets of input-output-outcome / impact indicators for each action within each priority area (equalling a total of 57 indicators), but the available indicators generally do not provide scope for this type of approach. For some indicators, more does not necessarily equal better. As such, the optimum value for Members States to achieve is not necessarily 100 per cent. Setting targets for each indicator is not within the scope of this study, but suitable targets need to be decided upon for some indicators (e.g. all indicators for priority 4, input and output indicators for priority 1). For several indicators, especially those in priority areas 2 and 6, factors such as size, native language or location of Member States can yield a natural advantage or disadvantage. Whether to establish ways of controlling for these (e.g. through different optimum scores) or to accept these differences as given in a large number of indicators is an issue worth considering Indicator selection one indicator per priority area It is possible in principle to reduce the three selected indicators per priority to one. To ensure consistency between the priority areas, these should be either the six input, output or outcome / impact indicators. The outcome / impact indicators are recommended for monitoring ERA progress because a lack of progress on any of them would suggest that ERA-related policies and initiatives may not be having their desired effect and warrant further investigation. The final set of selected indicators is provided in Table Table 3.10 Indicator selection one indicator per priority area Priority Priority 1: more effective national research systems Priority 2: optimal transnational cooperation and competition Priority 3: Open Labour Market for Researchers Priority 4: Gender equality and gender mainstreaming in research Priority 5: Optimal access to and circulation and transfer of scientific knowledge International dimension outside ERA (Priority 6) Indicator Number of patents per total public R&D expenditure Cross-border ownership of patents Proportion of doctoral candidates with citizenship of another EU MS Alternative: Researchers working in the business sector Proportion of women in grade A (professor) positions Public / private co-publication per million of the population Licence and patent revenues from abroad as a per cent of GDP Though this approach is straightforward and user-friendly, such a minimalist selection of indicators is problematic because it is difficult to represent progress on what are very broad ambitions by reference to a single, narrow indicator. For example, selecting input, output and outcome / impact indicators for each priority area can provide insight into whether resources have been invested in each priority, benefits are observable and elements of wider impacts are observable, resulting in an overall ability to assess effectiveness in fulfilling ERA priorities. Limiting indicators creates difficulties in terms of attributing positive effects to efforts made. As such, the one outcome / impact indicator per ERA priority represents the 2 April

40 minimum possible selection, but a set of three indicators across inputs, outputs and outcomes / impacts per priority provides greater insight into ERA progress Data collection needs and opportunities The analysis conducted of indicators across ERA priorities demonstrated that Eurostat, OECD, the IU Scoreboard, She Figures, MORE2, and the ERA Survey provide information that can be used for ERA progress monitoring. There are gaps in some areas but these do not suggest the need for entirely new data collection exercises. The study team has highlighted data collection needs across each priority area. This is primarily a task of refining the ERA Survey, and working on approaches to achieve better response rates on specific questions. This also includes the need for other sources to obtain data on some indicators more regularly, as is the case with non-eu doctorate holders through Eurostat. With a full set of complete and comprehensively updated data, composite indicators may be a future possibility, but ultimately the variation among available indicators highlighted in this study does not suggest this is currently a realistic option. 2 April

41 4 The role of national roadmaps in achieving the ERA Member States vary in their levels of ERA attainment, ambitions and resources, and in the research systems scale and structure. National roadmaps that set country-specific paths to ERA, and which define milestones and timelines, have a potentially important role in the ERA monitoring framework. This task provides an assessment of that role and how roadmaps could complement other components of the ERA monitoring system. The analysis considered aspects of the roadmap specification relevant to the terms of reference. This includes goal-setting, reporting (including report structure) and review processes. This section sets out and reviews current efforts to design and implement national roadmaps for achieving ERA objectives. It sets out the current political framework at EU level, describes efforts and ambitions of national roadmaps and activities at Member State level and suggests options to make the development of national roadmaps an integral part of the ERA monitoring mechanism. The analysis suggests that for national roadmaps to provide added value to existing monitoring activities, they should have a common structure whilst allowing Member States flexibility in pace and scope of action. Advice and guidance on the structure could be provided as part of the upcoming EU roadmap on ERA, which is currently being prepared by ERAC in cooperation with the European Commission. A draft template for national roadmaps is provided in A9.2, which could be complemented by further commentary and guidance developed in partnership between the Commission, Member States and stakeholders. 4.1 Political context at European level and steps towards a European roadmap The Lisbon Treaty establishes a legal basis for the ERA as a shared responsibility between the European Commission and Member States. 12 This has expanded the options and competencies for legislative action at EU level. The Commission has since 2013 included research actions in the country-specific recommendations as part of the European Semester, which is a review mechanism on Member States implementation of the EU s economic rules. In 2013, Members of the European Parliament (EP) also called for more binding legal measures at EU level to speed up completion of the ERA (EP, 2013). Member States insist on having autonomous national strategies (ERAC, 2014). Plans for Member States to develop an EU roadmap in cooperation with the European Commission were proposed by the Competitiveness Council in 2014 (Council of the European Union, 2014). An ERA roadmap at EU-level is expected in 2015, which will facilitate and complement efforts at national level. ERAC has repeatedly emphasised that Member State ownership and action should be at the centre of ERA progress and taken into account in developing an EU roadmap. The development of the EU roadmap should therefore be guided by the principle of shared responsibility and be built on contributions from Member States and the Commission. ERAC has set out the following principles, which the Committee agrees should guide the development of an EU roadmap: (ERAC, 2014b) emphasise implementation along existing priority areas; build on existing work by ERA groups (ESFRI, SFIC, High-level Group of Joint Programming (GPC)) and stakeholders (Science Europe, European University Association (EUA)); and concentrate on joint understanding of goals and no prescription of goals. The roadmap should set objectives for significant improvements in specific priority areas by 2020 and inform future monitoring exercises. The Competitiveness Council has selected a 12 Consolidated version of the Treaty on the functioning of the European Union, Article April

42 number of ERA actions where significant improvements can still be made. (Council of the European Union, 2014) These include: the use of open calls for proposals based on international peer review; the progress made by ESFRI and efforts to prioritise projects in the ESFRI roadmap; aligning, where possible, national strategies and research programmes with the strategic research agendas developed within the Joint Programme Initiatives to cope with major societal challenges; promoting wider uptake of innovative doctoral training principles; using open, transparent and merit-based recruitment practices; fostering mobility of researchers, and in particular between academia and industry; mainstreaming gender equality and the gender dimension in R&D policies and programmes; supporting open access to scientific publications and developing an effective approach for research data; promoting and implementing e-infrastructures; and fostering effective knowledge transfer in research and innovation between the public and private sectors. 4.2 State of play in Member States Germany published a national roadmap in July Poland and France are in the preparatory stages of developing their own national roadmaps. There are also some national roadmaps for individual action lines (e.g. research infrastructures). 14 Germany s roadmap includes a set of specific guidelines on how ERA should be further developed, which set the context and framework for the roadmap and can be understood as a set of recommendations directed at EU policy-makers, and specifically at the Commission. The guidelines highlight four aspects of future ERA development: the specific context of Germany as having a strong science and technology foundation for the further development of ERA based on scientific and technological excellence; the diversity and respective strengths of the individual national research systems: in line with opinions expressed in ERAC, the German guidelines stress diversity and commitment by Member States as drivers and oppose any legal harmonisation at EUlevel or proposition of specific regulation by the Commission; improving the effectiveness of the ERA and closing performance gaps between Member States through a combination of excellence-driven research funding and complementary sources, like the European Structural and Investment Funds (ESIF); and strengthening the international dimension of ERA with regard to third countries. The roadmap sets out problem areas, challenges, and a catalogue of objectives and measures across the six ERA priority areas. It takes specific account of the German research system (i.e. a federal system with shared competencies between federal government and regional government), its variety of actors and specific market and industry structure. Quantitative targets were only identified for priority areas 2.1 (transnational 13 Bundesministerium fuer Bildung und Forschung (2014), Strategy of the Federal Government on the European Research Area (ERA). 14 A list of national roadmaps for research infrastructures can be found here: (accessed 2 February 2015). 2 April

43 cooperation) and 4 (gender equality and gender mainstreaming). Under priority area 2.1, the Federal Government aims for a 20 per cent participation rate by foreign partners in projects funded by the Ministry of Education and Research. Under priority area 4, the Federal Government aims to achieve a proportion of women in scientific executive committees of at least 30 per cent. Information on how progress will be reported and final outcomes will be reviewed is provided only partially for priority areas 1 (more effective national research systems) and 2.2 (research infrastructures). Under priority area 1, the Excellence Initiative, a programme funding first-rate, internationally visible research, will be evaluated by an external expert group and a progress report on the Initiative will be presented. Both reports are expected in Under priority area 2.2, the roadmap document indicates that regular evaluations need to be performed both at strategic and operational levels. The document does not provide an integrated set of quantitative targets or information on how overall progress will be reported and reviewed. A report on progress is however expected to be presented to parliament (Bundestag) at the end of the legislative period in An overview of the German roadmap on ERA is provided in Annex 9, Table A9.1. In many priority areas, the German roadmap starts from a baseline of having an already well-developed institutional setup. Germany also has a variety of funding and financing instruments in place to cover most areas addressed by ERA priorities. Overall, the structure and format of the German roadmap could be used to inform an EU roadmap and guidance material on the development of national roadmaps in other countries. But the document displays clear shortcomings regarding goal setting and progress monitoring. 4.3 Potential role of national roadmaps in the ERA monitoring framework In principle, roadmaps can be useful tools for strategic planning and stakeholder buy-in. National roadmaps on ERA could be a viable tool to show pathways towards progress, define thematic priorities and showcase actions that take account of the specifics of national research systems, in particular in areas where indicators might not be available or the comparability across national systems does not enable reliable conclusions to be drawn from quantitative data alone. In these cases, roadmaps are tools to progress in a specific area through self-commitment. National roadmaps could complement the ERA monitoring mechanism and in particular the EU roadmap by showcasing individual pathways to progress. National roadmaps could also be presented and reviewed in an extended peer review exercise, building on mutual learning activities already performed in the context of ERAC. Experiences with ERA-NETs, Joint Technology Initiatives and Article 185 initiatives suggest that striking the right balance between developing a standard model and flexibility within the model are crucial to preventing a fragmented landscape and preserving flexibility that accounts for national specificities. For example, the Impact Assessment on Joint Programming Initiative and first interim evaluations of Joint Technology Initiatives in the field of Information and Communication Technologies identified a clear benefit in allowing for joint agenda-setting in the field of R&D on a voluntary basis (Goetzeler et al., 2008). 2 April

44 5 Role of peer reviews in ERA This task considered the potential role of complementary approaches to support monitoring and performance management mechanisms in the ERA. The study team examined the potential role of expert peer review mechanisms through which experts meet to consider how experience of other countries can help address particular challenges faced by a Member State in advancing towards ERA goals. This task provides two examples of peer review and challenge mechanisms from international and EU policy and considers whether and where such mechanisms might be applied to complement the ERA core indicators and national roadmaps. This section summaries the results of two case study reports (provided in full in Annex 11) and considers the potential role of Member State peer review in encouraging and monitoring ERA progress. 5.1 An introduction to peer review Peer reviews have been used to appraise the research and innovation systems of Member States in ERA since 2006, however the structures and the processes behind the peer reviews have varied. Member States would benefit from a more structured approach, supported by appropriate administrative and thematic assistance and embedded in a wider monitoring mechanism based on an EU ERA roadmap and national strategies on completing ERA. Peer review can be considered to be a specific form of mutual learning. The main types of activity conducted in country-level peer review processes are: fact-finding missions and expert visits by independent bodies and individual experts, which carry out on-site missions to investigate specific events and establish facts; and reporting and data collection including periodic reporting by participating countries to independent bodies or expert groups who analyse submitted reports. The OECD describes peer review as an examination of one state s performance or practices in a particular area by other states [ ] (Pagani, 2002) with the ultimate goal of improving policy making in the reviewed state and complying with established international standards and principles. Peer reviews rely heavily on mutual trust amongst the countries involved and the confidence of individual actors/representatives in the process. As a mechanism for learning and driving change, peer review can: contribute to a better understanding of Member State policies and identify transferable practice; improve efficiency and effectiveness of policies and implementation strategies; and facilitate the transfer of key aspects of policies, institutional arrangements or funding arrangements that have proven effective and efficient in their original context and are relevant to other contexts. In an ideal scenario, peer review provides positive peer pressure and incentives for reform through: formal recommendations and informal dialogue between peer countries; creating a space for public scrutiny, comparisons and ranking amongst countries; and creating impact on domestic public opinion, national administrations and policy-makers. The effectiveness of peer review depends on factors that include (Pagani, 2002): Value sharing amongst participating countries: 2 April

45 Agreement amongst the participating countries on the standards or criteria against which to evaluate performance. Common understanding is needed to prevent uncertainty or backtracking during the process. Adequate level of commitment by participating countries: Both human and financial resources should be provided at an adequate level to allow sufficient capacity for conducting the process. All participating countries should be engaged as examiners, as active members of the collective body or committee, and as subjects of the examination. Mutual trust amongst participating countries: A certain degree of trust and value sharing amongst the participants should be present from the beginning to facilitate the disclosure of data, information and essential documentation. Trust can be built throughout the process. Credibility and ownership of the process: The credibility of the peer review process is essential to its effectiveness and attractiveness compared to other expert studies or policy reviews. A strong linkage can be observed between the credibility of the process and its capacity to influence. Ownership of the process should be ensured by involving participating countries both in the structural design and the management of the peer reviews. Peer review mechanisms commonly require administrative and scientific support including the provision of a data repository in the form of a website or similar, administrative support to ensure comparable formats for peer reports, fact-finding missions and the synthesis of main outcomes in the form of seminars or workshops. 5.2 Previous peer reviews in the context of the ERA There has been some prior use of peer review in the context of the ERA. Ten peer reviews of nine different countries have been conducted; a comprehensive list of peer reviews conducted in relation to the ERA is provided in Annex This section analyses the approach and method, the nature of recommendations produced in ERA peer reviews and lessons for future ERA peer reviews Peer review practice in ERA A first round of peer reviews was conducted prior to 2008 by the European Union committee for scientific and technical research (CREST). These focused on reviewing the national policy mix 16 (EC, 2009b) of research and innovation policy and aimed at assessing national progress against the target of spending three per cent of the EU s GDP on research, as agreed during the Barcelona European Council in 2002 (European Council, 2002; EC, 2005). The positive role that peer review could play in supporting the modernisation of national research and innovation systems was noted by the Innovation Union initiative (EC, 2010). ERAC (formerly CREST) conducted a second round of peer reviews from 2010 that built on the Innovation Union Self-Assessment Tool (SAT), which provides a structure for Member States to assess their innovation performance. The SAT consists of 10 criteria that can be investigated by national ministries as part of a qualitative and quantitative self-assessment (EC, 2010): 15 ERAC has also conducted three Mutual Learning Seminars in 2012, 2013 and The purpose of these events was to exchange and document experiences with current practices from other Member States while advancing towards ERA goals (Tsipouri, Georghiou and Lilischkis, 2013; European Council, 2014). 16 The conceptual model of policy mix refers to the idea that it is the combination of policy instruments interacting among each other [ ] which influences R&D, rather than instruments taken in isolation (European Commission, 2009b). 2 April

46 promoting research and innovation is considered to be an important policy instrument to enhance competitiveness and job creation, address major societal challenges and improve quality of life, and is communicated as such to the public; design and implementation of research and innovation policies is steered at the highest political level and based on a multi-annual strategy; policies and instruments are targeted at exploiting current or emerging national/regional strengths within an EU context ( smart specialisation ); innovation policy is pursued in a broad sense going beyond technological research and its applications; there is adequate and predictable public investment in research and innovation focused in particular on stimulating private investment; excellence is a key criterion for research and education policy; education and training systems provide the right mix of skills; partnerships between higher education institutes, research centres and businesses, at regional, national and international level, are actively promoted; framework conditions promote business investment in R&D, entrepreneurship and innovation; public support to research and innovation in businesses is simple, easy to access, and high quality; and the public sector itself is a driver of innovation. The second round of peer reviews included five reviews of national research and innovation systems based on the SAT, supported by Commission services and external consultants. Three SAT pilots were conducted, reviewing Belgium, Estonia and Denmark from Two further reviews, on Spain and Iceland, were conducted on the basis of the three pilots. The peer reviews recommended strategic actions on governance of the research and innovation system or planning and prioritisation of high-level themes (e.g. stronger focus on public-private cooperation in research, stronger support for university spin-offs). Very specific recommendations can be found in the final report of the Spanish research and innovation system. These included the establishment of a monitoring and evaluation system using specific indicators proposed by the French peers involved in the review (Spanish Ministry of Economy and Competitiveness, 2014). There is no conclusive evidence of how national ministries responded to the recommendations and whether the recommendations were taken into account in subsequent reforms of the national research and innovation systems. The process used for peer reviews built on the Innovation Union s Self-Assessment Tool since 2010 is shown in Figure April

47 Figure 5.1 Country-to-country peer review process in ERAC using the Innovation Union Self- Assessment Tool Source: ICF International, based on Crasemann et al. (2012) and Rambøll (2012) A workshop held in 2012 (Rambøll, 2012) summarised findings and discussed the utility of the peer review tool based on SAT to monitor progress and identify challenges in completing the ERA. The individual peer reviews on Denmark, Belgium and Estonia also provide some insight into the overall process and initial feedback from government participants. Capacity to support further peer reviews might be available through the Policy Support Facility (PSF), which is currently being set up by the European Commission (EC, 2014b). It aims to improve the design, implementation and performance review of national and regional research and innovation policies, and to provide technical assistance and expert advice to government authorities at different levels. The exact structure and services offered by the PSF were not clear at the time of this study Lessons for the future of ERA The workshop and evidence from the final reports of the peer reviews on Denmark, Belgium and Estonia indicate that the ERAC peer-reviews based on the SAT have been viewed very positively and were considered to provide further structure to the peer review process under ERAC by participants and reviewed countries. Some lessons can therefore be drawn from past ERAC peer reviews to enable systematic and efficient peer reviews of national research and innovation policies in the future (Crasemann et al., 2012). The peer-review method based on the SAT provides a flexible approach in terms of thematic focus and organisation. The quality of the recommendations and evidence produced relies on the input and engagement of experts, government peers and stakeholders. Some issues listed in the SAT are very broad and systemic, whilst the political interest behind a peer review often is motivated by specific discussions on individual legislative items or funding programmes. There is limited integration with other policy fields (e.g. economy, employment, regional development) since the structure used is built on the SAT Tool, which focuses on 2 April

48 research and innovation policy. The SAT might neglect or miss essential topics and issues that are addressed within the national research and innovation system. The peer reviews conducted so far seem to be considered as singular activities. There is no clear and simple process for Member States to express interest and have peer reviews conducted. 5.3 Use of peer review in other policy areas Country-to-country peer review and mutual learning mechanisms are in use in other areas of EU policy and are used by other multi-lateral institutions. Two examples from the areas of education and employment policy (in the OECD and EU) and one from EU agri-food policy have been analysed to provide insight into how peer review mechanisms can be used. They explain the peer review mechanism adopted and its concept, and identify lessons relevant to the ERA. These inform recommendations provided in section The case studies were selected because they are well-established peer review mechanisms. The education and employment examples have thematic relevance in relation to research policy, as well as structural similarities (e.g. similar actors involved) and similarities in the overall progress monitoring arrangements (i.e. similar complementary monitoring instruments are available). Detailed case studies for the education and employment case studies can be found in Annex OECD peer reviews in education and training No other international organisation has used peer review as extensively as the OECD. The OECD has an established peer review practice, which is used in many policy areas, including education and learning. The reviews entail systematic examination and assessment of a member country s performance by other member countries, with the ultimate goal of helping the reviewed country improve its policy making, adopt best practices, and comply with established standards and principles (Pagani, 2002). The peer review process in the area of education typically involves the following elements: basis for proceeding, that is, decisions by the Education Policy Committee (EDPC, the main policy-making body in OECD education policy), programmes agreed at ministerial level or provisions in treaties and other legally binding documents; an agreed set of principles, standards and criteria against which country performance is assessed (e.g. policy recommendations and guidelines, specific metrics, indicators or legally binding qualitative or normative principles); designated actors to carry out the peer review: this typically includes the reviewed country, the examiner or peer countries, the EDPC as a collective body and the OECD secretariat which provides administrative support; and a set of procedures leading to the final result and publishable material. The examination is conducted on an advisory basis and the entire review mechanism is based on mutual understanding 17 of the countries involved in the review. Typically the peer reviews are structured in three phases: The preparatory phase involves review of background documents, self-assessment by the country under review, preparation of documentation, guidance material, questionnaires and data provided by the OECD secretariat. 17 Mutual understanding refers to a basis for proceeding; an agreed set of principles, standards and criteria against which the country s performance will be reviewed; designated actors to carry out the review; and a set of procedures leading to the final result. 2 April

49 The consultation phase consists of peer countries and the OECD secretariat reviewing questionnaire responses and other data, carrying out on-site visits, consulting with interest groups, civil society and academics. The OECD secretariat prepares the draft final report and shares it with peer countries and the reviewed country. During the assessment phase the draft final report is discussed in the EDPC, and receives final amendments by peer countries or other delegates (e.g. other country representatives, non-governmental organisations) and is adopted by EDPC. The final report is published and disseminated through a press release. By providing structured administrative support and working to an agreed and transparent set of rules and processes, OECD peer reviews create a system of mutual accountability (OECD, n.d.). The OECD process benefits from an established institutional set-up built on strong administrative support from the OECD secretariat and a well-defined set of principles, standards and procedures. Peer reviews in the area of education policy are available on the OECD website (OECD, n.d.) Lessons for the future of ERA The following lessons for the use of peer review on the ERA can be drawn from OECD experiences in the area of education and training: a secretariat can be used to reduce the burden of evidence-gathering and background research on the reviewers, allowing experts to make more effective use of their time; Member State involvement in the development of guiding principles, procedural arrangements and resource commitments is critical to ensuring ownership and credibility of the peer review mechanism; continuous moderation of the process and agreed targets as well as established and commonly accepted indicators provide for greater comparability and transparency of individual peer reviews; peer reviews only involving government officials and OECD staff may be biased regarding the formulated recommendations and allow only for a limited level of criticism; peer review processes can contribute to improvements in the host country and to the definition of good practice for the community at large; and publication of the results provides transparency EU employment policy The European Employment Strategy (EES) is part of the EU's growth strategy (EC, 2014d). The EES aims to create more and better jobs throughout the EU. A system of peer reviews linked to the EES was set up in A revised Mutual Learning Programme (MLP) was launched in 2005 in response to a request from the European Council to develop more robust and integrated approaches to mutual learning in the area of employment policies. MLP activities are aligned to the EU s annual cycle of economic policy guidance (i.e. the European Semester), including the European Semester s specific work programme and budget as well as the Country Specific Recommendations issued by the European Commission each year. The MLP is managed by the European Commission s Directorate- General for Employment, which is supported by an external contractor to implement the programme. The MLP includes a range of activities such as thematic events, peer reviews, learning exchanges and targeted dissemination events. The MLP also manages a database of labour market practice containing examples of effective policies and measures within the context of the EES. 2 April

50 Peer reviews are used to assess the implementation of concrete policy measures on a given topic. Peer reviews allow the exchange of experiences between a host country and peer countries. The host country presents, and wishes to gain feedback on an effective policy (and associated good practice). Peer countries are interested in learning from the host example and potentially transferring it into their national setting and in sharing their own policy experiences with the host and other participating countries. Thematic reviews are one specific type of event organised during the first semester of the year before the negotiation of new Country Specific Recommendations. Each thematic review focusing on a thematic grouping of Country Specific Recommendations follows a common format in which Member States act as both reviewer and reviewee. Reviews focus on recent policy developments introduced by Member States since the last review. The MLP is based on a tested and tried model, largely approved by participants and enjoying continuous interest from Member States. Research currently undertaken by ICF 18 suggests that approval of the MLP is largely due to: the high political and public pressure for action in the wake of the economic crisis; a formalised progress measurement system (European Semester, National Reform Plans and Country Specific Recommendations as described above); and a dedicated administrative and technical support mechanism provided through an external contractor. In this context, the MLP exerts an indirect influence on national policy making Lessons for the future of ERA The following lessons for the use of peer review on the ERA can be drawn from EES experiences in the area of employment policy: a main strength of the EES Mutual Learning Programme is that the activities are run as a programme with their own annual cycle closely aligned with the European Semester s work programme; the formulation of national guidance and strategy including a corresponding feedback mechanism at EU level through the European Semester have provided added-value to the EES Mutual Learning Programme; the availability of sufficient budget to provide for the external support service is a critical success factor of the EES Mutual Learning Programme; and the annual summary of peer reviews and thematic reviews conducted under the EES Mutual Learning Programme, provided for in the Multilateral Surveillance Conclusions, helps to strengthen ownership of the MLP in Member States, and increases peerpressure and mutual accountability between Member States. Introducing a structured mutual learning programme in ERA could improve convergence between Member States and bridge certain gaps in effective monitoring that the ERA monitoring mechanism is currently lacking EU Task Force on the eradication of animal diseases A Task Force on the eradication of animal diseases was created in 2000 following recommendations made in the White Paper on Food Safety. Its specific objectives were to improve animal disease eradication and the cost-benefit ratio of animal disease eradication programmes co-financed by the Community (EC, 2012). It is maintained to assist Member 18 Study commissioned by the European Commission on comparison and assessment of the effectiveness and efficiency of different OMCs to propose innovative governance methods in the ET 2020 context. 2 April

51 States to implement animal disease eradication programmes suitable for meeting set objectives and allowing harmonisation of programmes across different Member States. Task Force meetings are held in Brussels once or twice per year. They are attended by representatives of the 28 Member States, the chairpersons of the Task Force s expert subgroups, and by Commission representatives (EC, 2012). A Commission official chairs the meetings and membership is generally stable and consistent from meeting to meeting. All aspects of the programmes can be discussed. The findings and recommendations of the subgroups are delivered and presentations made relating to topics relevant to all diseases subject to EU-funded measures. Subgroups of thematic experts have been established for specific animal diseases. The Task Force sub-groups feed information back to the plenary Task Force on the state of programmes in individual Member States, including any specific problems they face. Subgroup meetings take place in Member States where a programme for disease control and eradication is being approved for co-financing by the Commission. The situation in that country is presented and the subgroup puts forward advice on how it might be improved. The chair and experts draft recommendations, and Member States are informed. As the subgroups include representatives from other Member States, experience on successful measures elsewhere can be put forward for consideration and serve as a basis for recommendations. As the meetings take place within Member States, representatives from public and private sector industries involved in these programmes can be present and get involved. Research conducted by ICF for the Commission suggests that Member States consider the Task Force to be a useful mechanism that has brought added value to their programmes. Research has found that national authorities value Task Force sub-groups input to the design of programme plans. Measures are often technical and require precise implementation procedures to be followed in order to be effective. Failure to observe correct procedure can reduce the effectiveness of a measure, potentially undermining the effectiveness of the programme. Task Force experts provide detailed and practical advice about which measures should be implemented and most importantly, in the opinion of the stakeholders, advice on how the measures should be implemented. In addition to technical support, the Task Force can provide host authorities with the support to implement measures deemed necessary. The Task Force members were perceived to be external experts independent of any particular agenda. This was considered to be especially useful by national authorities when they were attempting to build consensus for novel or revised measures. Drawing on the expertise of external experts with a track record in successful programmes assisted the national authority in justifying proposed measures Lessons for the future of ERA The following lessons for the use of peer review on the ERA can be drawn from Commission experiences in the area of eradicating animal diseases: the breadth of expertise of the visiting review team affects added-value and impact so it is important to recruit the right people to the task; the independence of external experts was considered very useful by national authorities subject to reviews; visits from the Task Force provide the opportunity to subject programmes to critical challenge from recognised independent experts and can help in building consensus for novel or revised measures; the manner in which the review is conducted is important: it needs to provide critical challenge but also be supportive and collaborative, and avoid an audit or inquisitorial approach; 2 April

52 the peer review process can help to empower those in the recipient country who are working for change; and findings from individual reviews can be combined to produce guidance of relevance to the EU as a whole. 2 April

53 6 Conclusions and recommendations Following the development of the methodological framework for the evaluation and publication of the 2014 ERA Progress Report by DG RTD, the study team identified practical and theoretical difficulties with the current ERA progress monitoring framework. Together with feedback from Member States (e.g. responses to the Second Progress Report), this suggested that there was a case for reviewing the current approach to see where and how it could be improved. For many ERA priority areas there is no agreed quantitative goal or target against which to measure Member States situation or progress. Success, or minimum acceptable performance, is undefined. In the absence of such goals the progress reports have measured Member State performance with reference to an EU average. This approach provides information about the situation across Member States but not whether the ERA has been completed in that priority area. Setting goals and targets would enable an assessment of progress made towards a desired goal for the ERA. But this kind of forward-looking assessment cannot be made without defining the desired or expected end-result for the ERA, i.e. the completion of the ERA, when the strategic objective will have been achieved. The current monitoring framework would also benefit from greater clarity about overall ERA system objectives and established goals for the system to meet. The lack of established goals for the indicators and ultimately for the actions and priorities means that if the evaluation had been undertaken, it would only have been able to describe progress in relative rather than absolute terms that is, it would enable an assessment of whether there had been change from one period to the next but not whether individual countries and the EU as a whole were making progress towards an overall objective. Other issues identified include: The current progress reporting system has been criticised by Member States for not being able to recognise the diversity of Member States research systems and pathways towards achieving ERA objectives. Each Member State is starting from a different baseline, so while Member States results for a given indicator are being reported and can be compared, in the absence of contextual information about the size or structure of the Member State research systems it is difficult to interpret the results both at the Member State level and in terms of the implications for the EU as a whole. The current system often documents activity such as introducing new policy measures in the domain of the ERA-priorities rather than measuring progress in terms of outcomes and impacts. The current reporting arrangements capture information on Member State actions and activities that have relevance to a particular ERA priority area, but it is often not clear what kind of impact the action/activity has had or will have on progress towards ERA objectives. Securing robust information to inform decision-making has proven difficult. In 2013 and 2014 a survey instrument was distributed to RPOs and RFOs. Response rates varied widely and for many Member States were too low for the results to be regarded as representative. 19 And without contextual information on the size and structure of the Member State research system, it can be difficult to interpret the results. 19 There are few registers in the statistical offices of MS where RFO's and RPO's are included. For this reason the European Commission s Directorate-General Education and Culture (DG EAC) has introduced the European Tertiary Education Register (ETER), a database of HEIs in Europe, currently including 36 countries and 2,673 2 April

54 ERA progress reporting where additional data are required adds to the administrative burdens on governments, RPOs, and RFOs in the Member States. Therefore, existing data should be used as much as possible. ERA progress reporting sits alongside other information gathering and reporting mechanisms such as the National Reform Programmes and the European Semester. The indicator set has been unstable due to changes in the composition of the set of indicators. ERA progress reporting could better support and link with other related initiatives, particularly the European Semester and Innovation Union. It is difficult to gauge progress, even within the constraints listed above, because the ERA indicators adopted have changed from year-to-year and comparable data bridging a two-year period (or longer) are not available for most indicators. The frequency and timing of ERA reporting could be better tuned to the pace of change, the availability of data and the requirements of users. It also could be better aligned with the European Semester and the Innovation Union. 6.1 Indicators and suitable metrics to measure progress The indicator appraisal conducted identified available data, geographic coverage and data gaps, particularly for outputs and long-term impacts, which can be clearly linked to activities and outputs under the individual ERA priorities. Across the different ERA priorities, output and outcome/impact indicators are either difficult to identify or, more often, overlap across individual inputs. Problems in linking individual inputs to specific, long-term impacts are visible across all priority areas. There are also issues related to the thematic breadth of some priority areas (e.g. priority area 3 open labour market for researchers), where outputs are very difficult to measure across all actions. The study team has proposed a set of indicators, with one input, output and outcome / impact indicator for each priority area, which reflect the best available data and are most relevant to a given priority. Data availability can be improved in the future, and the team has identified other indicators that could be used in the future where this is the case. The indicator appraisal also showed a clear difference in Member State ownership between data collection undertaken for the ERA and the Innovation Union Scoreboard. The indicators proposed and future data collection should therefore be better integrated with work already undertaken for the Innovation Union Scoreboard. Efforts should be made to reduce the administrative burden as much as possible for Member States and relevant stakeholders (RFOs and RPOs). 6.2 Desirable attributes of a future evaluation and monitoring system The following attributes of an ERA evaluation and monitoring system have been identified as particularly appealing by Member States and the Commission and should be considered in future initiatives: supporting performance management by focusing on identifying important gaps, securing commitments to close them and tracking progress; recognising the diversity of Member State research systems and pathways to ERA; ensuring that indicators remain stable between reporting periods (and, as far as possible, resilient to adjustments in the ERA priorities); HEIs; 29 countries have provided a full set of data for 2,250 HEIs. ETER presents comparable information under headings such as numbers of staff and students, subject domains covered, research activity and expenditures. 2 April

55 using robust indicators that are relevant to ERA priorities and which are based on reliable and already-existing information as far as possible; being proportionate and efficient, for example, by being well-integrated with other information and analytical systems, minimising additional administrative burdens and reporting at appropriate frequencies; and integration with other relevant policy initiatives and reporting mechanisms (especially the Innovation Union). 6.3 Towards an integrated approach to measuring ERA progress Current monitoring arrangements and suitable indicators identified by the study team only cover parts of the overall activities and link some inputs, outputs and impacts in each ERA priority area. They also require complementary approaches to reflect Member State diversity and encourage progress reporting. Building on the work already undertaken by ERAC, a future integrated approach could include individual national roadmaps to allow for specific focus on strengths and weaknesses of national systems and iterative performance review against a core set of indicators accompanied by individual national objectives set out in roadmaps Ensuring added value of national roadmaps A draft of the EU-level ERA roadmap is expected to be discussed in The recommendations developed in this study therefore aim to inform the discussion and development of robust Member State roadmaps. National ERA roadmaps could provide for flexible target setting, taking into account national specificities, if designed appropriately and recognised by Member States as tools for achieving ERA progress. The research undertaken for this study found that ERAC and the Commission should aim at informing certain framework conditions in developing guidance and advice on national roadmaps as part of the European ERA roadmap. This would include a set of non-binding recommendations and aim to establish a high level of mutual trust between Member States and the Commission through clear and transparent discussion on and review of national roadmaps within the context of ERAC. To provide a future monitoring and progress measurement framework with national targets and pathways towards completion of ERA, national roadmaps should however meet a number of principal criteria that a European roadmap could set out, including: clearly identifying relevant actors within the national research system and their role in achieving progress; a clear definition of specific problem areas and challenges which prevent the Member State from implementing the ERA; quantitative targets and actors responsible to reach these targets alongside the priority areas proposed for an EU roadmap; a timeline which plots individual activities and targets against a set timeframe; a mechanism for progress reporting, including the actors responsible for progress reports; and a mechanism to review progress on a regular basis, including the actors that need to be involved and planned frequency of review. Further deliberation between the Commission and ERAC should reflect on the following aspects: the necessary level of robustness and detail of national roadmaps; 2 April

56 resources that can realistically be invested at national level in design, monitoring and reporting on national frameworks; and how flexibility for Member States can be preserved Recommendations for future use of peer review and mutual learning in ERA Study team analysis of the use of peer review in the ERA to date and the case study examples suggest that a peer review mechanism could have a positive role to play in completing the ERA if appropriately constructed, operated and resourced. Future ERA peer reviews could be organised as a joint effort by the Commission and Member States, built on the concept of shared responsibility (TFEU Article 182.5) (EU, 2012). Member States can be encouraged to initiate the development of a common approach and guidelines to establish a formalised and credible peer review system. Available evidence also suggests that it can be helpful to identify a group responsible for driving progress and identifying where progress is lagging. These could be either technical experts on specific ERA priorities, an independent support service, or staff responsible for supporting ERAC. The planned Policy Support Facility (PSF) could serve as a hub for strategic intelligence and provide administrative support and guidance to policy makers, participants and interviewees. The peer review mechanism s prospects for success are improved if it incorporates the following elements and approaches: Decisions on the design and implementation of a peer review programme and approval of the outputs being taken by ERAC. The peer review process needs to be properly documented and well understood by Member States. Member States need to be centrally involved in the development of guiding principles and procedural arrangements. The Innovation Union self-assessment tool already in use provides a starting point. The process should be constructed and operated in a manner that engenders trust, collaboration and openness. Peer reviews should provide critical challenge but also be supportive and collaborative. The scope of reviews and selection of reviewers could be organised based on the principal ERA objectives or individual measures identified in the EU ERA roadmap in A structured approach is needed to develop national roadmaps and a corresponding EU feedback mechanism potentially linking into the European Semester. The peer review teams should not be dominated by Member State policy administrators but rather should include a mix of experts, including independent experts. A more structured approach towards the provision of thematic and administrative support and developing guidelines, templates and a structured knowledge management system for collected data and analytical reports might encourage more Member States to express an interest in being peer-reviewed. An annual summary of peer reviews, drafted by ERAC (i.e. by the Member States themselves) would strengthen ownership, provide guidance for the EU as a whole and build mutual accountability between Member States. The SAT might neglect or miss essential topics and issues that are addressed within the national research and innovation system. A review of the SAT and the inclusion of relevant overlapping policy fields might increase interest from Member States; 2 April

57 The amount of preparatory material should be kept to a minimum to reduce administrative burden of participating organisations. Wider organisational learning effects can be achieved if the peer review results are disseminated widely across relevant stakeholders in all EU Member States and Associated Countries. 6.4 An illustrative rating system for progress reporting Progress and performance at EU level could be demonstrated across ERA priorities through the use of a traffic light system similar to that used in the British Civil Service to track performance. This could signal the status of progress towards ERA objectives and EU level goals. Progress could be assessed against the goals defined under the EU roadmap. Progress could be assessed against all ERA priorities or specific areas of action identified in the roadmap. An example of how this approach could be used in the ERA is provided in Table 6.1. Table 6.1 Traffic light rating system for ERA progress Performance Description Taking all relevant information into account, the aggregate assessment is that this aspect of the ERA is either complete or track for completion by the given deadline. The available information suggests that achievement of this aspect of the ERA within the given timetable is in doubt. There are issues to be addressed but the situation is not irrecoverable providing the risks are addressed. There are significant problems and achievement of this aspect of the ERA is not expected within the given timetable. Progress objectives are not being met and actions is required. Status This approach offers succinct and easily identifiable messages about the state-of-play at EU level on ERA priorities. Traffic light ratings could be assigned by a team of individual experts, who review data collected against the core indicator set and peer review reports. 2 April

58 ANNEXES 2 April

59 Annex 1 Study tasks and work completed against objectives The study terms of reference set out six main tasks for the study. Table A1.1 outlines the original tasks and the work undertaken by the study team to meet the study objectives for those tasks. Table A1.1 Study tasks and work completed against study objectives Task Original scope of work Status of original task for present study Collect data on national measures to fill gaps for the 2013 reporting period per ERA action and MS / AC Establish the baseline for AC in cases where this had not yet been done and revise and update country fiches for MS and AC with new or planned ERA-related measures Develop and estimate policy progress indicators for 2013 Develop the evaluation framework to assess progress Prepare a methodological note establishing the approach for evaluating progress between the 2013 and 2014 reporting periods Evaluate progress of ERA policies and actions Data gaps and missing information were addressed by DG RTD through consultation with the MS. Three ACs also provided new measures. The task was executed as originally planned. Policy progress indicators for the 2013 period were developed and estimated by DG RTD. An evaluation framework was developed and tested by the study team for the period. The evaluation could not be undertaken as originally envisioned. A methodological note was prepared as originally planned. The evaluation could not be completed as originally envisioned this task was replaced by the development of future evaluation framework. Work completed to meet study objectives Country experts from the study team reviewed and checked the new measures, provided comments on their validity and suggested changes to DG RTD for inclusion in the 2014 Progress Report. New measures were researched for those AC that did not provide new measures. This information was incorporated into the 2014 progress report. Country experts from the study team prepared new AC fiches. Existing MS and AC fiches were reviewed for consistency, formats were standardised and data gaps filled where possible. Country fiches are provided on the DG RTD ERA website (EC, 2014c). The study team reviewed indicators that were used in the 201 and 2014 reports and identified and appraised additional indicators that could be used instead of or to complement existing indicators. A new evaluation framework was developed for future progress evaluation. Study team analysis and proposals for a new framework are described in this report. Analysis arising from the preparation of the methodological note is provided as annexes to this report. The study team developed a framework for future evaluation and monitoring of the ERA, which is set out in this report. 2 April

60 Annex 2 Methodology to assess ERA progress The original aim of this study was to provide an evaluation on progress in the achievement of ERA in Member States and Associated Countries by reference to current and planned national policies in the evaluation period (July 2012 to August 2014) for each ERA action. Progress was to be assessed at EU level by ERA priority as defined in the ERA communication A Reinforced European Research Area Partnership for Excellence and Growth (EC, 2012) including actions derived from the Commission Communication Enhancing and focusing EU international cooperation in research and innovation: A strategic approach as deemed necessary (EC, 2012c). The study was intended to build on the first ERA-progress report (EC, 2013d) adopted by the Commission in September 2013 and work undertaken by DG RTD for the second ERA-progress report The study was expected to assess the implementation of ERA actions by research funding organisations, where relevant, but not assess implementation by research performing organisations. The expectation was to base the evaluation almost entirely on information collected for the 2013 and 2014 progress reports, since these provided the most complete assessment available of initiatives undertaken by Member States and Associated Countries. The evaluation was expected to use the progress indicators as well as other evaluation methods and tools, where appropriate. This annex explains the process that the study team developed to evaluate progress, the results of applying the first three steps of the evaluation methodology, the limits of the available data and tools, and gaps foreseen. Additional detail of the analysis is provided in Annex 3 - Annex 6. A2.1 Overview of the evaluation method proposed Our task was to evaluate the progress made since the first progress report towards the construction of the ERA by assessing: The measures and initiatives undertaken by Member States and Associated Countries; and Any changes in inputs, outputs and outcomes across the ERA indicators. The method was designed to accommodate the following sources of complexity: Changes were made between the 2013 and 2014 assessments to the set of indicators on which DG RTD has collected data. A set of indicators was identified for the first ERA progress report. These indicators were used to establish a baseline for measuring progress. The Methodological note: ERA Progress Indicators (EC, 2013) analysed the indicators used. The report provided suggestions for the reformulation of several indicators, and the inclusion of new or removal of indicators. As a result, some of the 2013 report indicators were excluded from the 2014 assessment, while others were reformulated. This means determination of progress since 2013 across the ERA as a whole cannot be achieved through a simple comparison of 2013 and 2014 indicator data. Qualitative information on new measures and initiatives in Member States and Associated Countries provides limited insight on progress. The study team could not determine from the qualitative information supplied by Member States and Associated Countries whether, and to what extent, the measure/initiative has been implemented. Nevertheless, this information could have been relied upon for the evaluation where comparable indicator data were lacking. A forward-looking progress assessment is not possible because there are no goals set for individual indicators or for actions/priorities. The progress assessment would have benefitted from having reference goals and objectives set for the indicators and actions/priorities. These would enable the review to comment on the progress made towards a desired end state, or goal, for the ERA. But this kind of forward-looking 2 April

61 assessment cannot be made without defining the desired or expected end-result for the ERA (i.e. when the strategic objective will have been achieved). The process proposed for the evaluation of progress is shown in Figure A2.1 and described below. Figure A2.1 Flow Diagram for ERA Evaluation Step 1: Scope test Check whether the indicators refer to implementation by RPOs. Are the indicators in scope? Step 2: Consistency test Compare progress indicators with indicators from the previous reporting period. Are the indicators the same? Step 3: Data availability test Check availability of data for new indicator in the previous reporting period. Are the data available for both periods? Indicators excluded from the study Step 4: Assessment of other information Review other available information Can any (qualified) judgement on progress be made? Step 3: Data availability test Check availability of data for previous indicator in the new reporting period. Are the data available for both periods? Progress cannot be assessed Step 5: Assess progress A2.2 Step 1: Scope test A2.2.1 Description of the scope test Indicators in both reporting periods were assessed to determine whether they were being used to demonstrate implementation of ERA actions by RPOs or by RFOs, by Member States or by others. Indicators that refer to ERA implementation by RPOs were excluded from the evaluation as they are out of scope. Further evaluation tests explained below were only applied to indicators within study scope. A2.2.2 Results of the scope test There were 33 indicators used in the first reporting period and 60 in the second reporting period. Of these, 22 indicators from the first reporting period and 35 from the second reporting period were carried to Step 2. Annex 3 provides a complete list of the 2013 and 2014 indicators, with an indication of whether they are within study scope. 2 April

62 A2.3 Step 2: Consistency test A2.3.1 Description of the consistency test Step 2 was a consistency test that compared the indicators within scope between the first and second reports to determine those that remained the same or changed. There are two possible outcomes for each indicator: 1. The indicator did not change. 2. The indicator changed. The outcome affects the test applied in the next step of the process. Where indicators had changed, the study team would have assessed whether the change was substantive or minor ; if the change was substantive, the indicators would be considered incomparable, but if they were minor, the study team checked whether comparison across the two periods was still possible so that those indicators could proceed to Step 3. A2.3.2 Results of the consistency test The consistency test showed that there were only six indicators that could have been directly used to assess progress, covering only three of five priority areas (one indicator each for priority areas one and three, and four indicators for priority area two). Annex 4 shows the indicators that remained unchanged between the two reporting periods, which could have been directly compared to assess progress. A2.4 Step 3: Data availability test A2.4.1 Description of the data availability test Step 3 was an assessment of data availability for each indicator. The specific test applied to each indicator varied according to the result of Step 2. A Indicator was unchanged The DG RTD ERA progress database was interrogated to determine whether it held data for both reporting periods for the indicator in question. Where data for both periods were available then the indicator progressed directly to Step 5. If not, then the indicator went to Step 4. The DG RTD database is populated with data from OECD reports, EUROSTAT, RFO surveys, ESFRI, She Figures and information provided by the Member States to DG RTD. A Indicator changed Where the indicator changed after the first reporting period, the ERA progress database and other sources were interrogated to determine whether data were available for both reporting periods for the old or the new or both iterations of the indicator. A2.4.2 Results of the data availability test Data sources for each indicator were assessed and an initial judgement made about the extent to which indicators that had changed could have been used where data from the first reporting period was available for new indicators or data from the second reporting period was available for old indicators. Some indicators changed between the first and second reporting period but matching pairs of information from the two periods were available (information was available in either the first reporting period for the new indicator or the second reporting period for the old indicator). An additional 37 indicators, detailed in Annex 5, Table A5.1- Table A5.5, might have been used to inform the progress assessment on this basis. The status of the indicator appraisal is summarised in Figure A April

63 Number of indicators Assessment of progress in achieving ERA - Final Report Figure A2.2 Summary of the indicator appraisal following the application of steps Test 2 Test 3 Total Changed Changed and subject to Test 4 Unchanged Changed but useable Source: ICF (2014) A summary of the results of steps 1-3 for each priority, by action and indicator is provided in Annex 3. It shows where progress could have potentially been assessed based on available information and where additional information would have been required to fill gaps. At this stage in the evaluation, the study team reviewed the data collected for the 2014 Progress Report and that collected in Indicators that relied on RFO survey data, particularly in 2014, were largely considered to be unusable for the evaluation due to low response rates from funding organisations and low comparability between respondents in 2013 and As most of the indicators were based on RFO survey data, this eliminated the majority of available indicators across the priority areas. This would have resulted in the following gaps in the analysis: No indicators for actions under priority 1 (one indicator could have been used related to outcomes); No indicators for actions two and three under priority 2 (one indicator could have been used for action one, two indicators for action four, and one outcome indicator); No indicators for actions under priority 3 (two outcome indicators could have been used); No indicators for actions under priority 4 (four outcome indicators could have been used); and No indicators under actions two and four under priority 5 (one indicator each for actions one and three could have been used). A2.5 Step 4: Assessment of other information Step 4 would have involved a review of information on new initiatives and measures provided by the Member States and Associated Countries to determine whether information was available that was relevant to specific actions that could have been used to judge progress between the first and second reporting periods. This information could have helped to fill gaps in assessing progress on ERA actions where comparable indicator information was unavailable and to broaden and contextualise the evaluation where indicator information was available. The information would have been used to indicate in qualitative terms whether a country had taken steps towards an ERA action where indicator information was unavailable. The study team expected a priori that the information available would not necessarily have enabled determination of whether the initiative had been implemented or whether there had been an impact on progress towards the priority. 2 April

64 Step 4 was not undertaken, however, because earlier review and input to the country fiches for the 2014 Progress Report revealed that, as expected, the available qualitative information was insufficient for most countries to determine whether the initiatives cited had been implemented or whether there was progress made. The available information was also inconsistent across countries and therefore largely incomparable. A2.6 Step 5: Assess progress The final step of the evaluation would have been a report on progress at two levels: Progress at Member States and Associated Countries level since the first reporting period. Overall EU progress against each ERA priority since the first reporting period. Reporting on overall progress at EU level would have depended in large part on the degree to which information on progress was (a) available and (b) comparable across Member States and Associated Countries. The study team found that the available information was limited. The 2009 Expert Group report on ERA indicators suggests that progress could incorporate available information on other science and technology indicators. The advantage of these indicators is their availability over a longer time period. They could therefore be used to put the ERA indicators into a broader perspective. Countries could also have been grouped based on common structural similarities, similar to the scoreboard approach taken by other monitoring exercises at EU level such as the industrial scoreboard or the IU scoreboard. They show how countries do from one year to the next based on a limited set of indicators. The scoreboards group countries into four categories. Accordingly, progress by actions and priorities could have been categorised by countries that are leaders, followers, catching-up, or lagging (these categories are illustrative). The changes could have been shown graphically rather than through quantitative figures or qualitative description to avoid suggesting that progress can be measured precisely. Figure A2.3 provides an illustration of a potential scoreboard approach. Figure A2.3 Example of a scorecard Source: Technopolis, adapted from EPSIS (2012) However, this step as described is only theoretical, since the evaluation did not proceed beyond Step 3, and the initial assessment of data availability under Step 4. A2.7 Summary of evaluation limits, risks and gaps This section provides a short commentary on issues that arose in preparing the evaluation framework. These were taken forward into the development of a future monitoring and evaluation framework. A2.7.1 Lack of comparable indicators The large number of changes made to the indicator set after the 2013 report means that if the evaluation had proceeded under Steps 4 and 5, reporting on progress in 2014 against the 2013 baseline would have been directly feasible for only a small number of indicators. 2 April

65 A2.7.2 Distortion effects arising from survey data Using the RFO survey results to report on progress would have been problematic for several reasons: there were numerous instances of responses from a small number of RFOs in a country or differences in the numbers of RFOs between reporting periods which would have resulted in (potentially dramatic) under- or over-reporting on progress in a country. One potential way to overcome this problem would have been to use additional, qualitative information to assess progress that complemented the survey data, but as reported above, the qualitative information available did not provide sufficient information (e.g. on implementation or progress achieved) and it was not sufficiently comparable across Member States for this purpose. A2.7.3 Reliance on relative, rather than absolute progress assessments It is also clear that the framework would benefit from greater clarity about overall ERA system objectives and established goals for the system to meet. The lack of established goals for the indicators and ultimately for the actions and priorities means that if the evaluation had been undertaken, it would only have been able to describe progress in relative rather than absolute terms. That is, it would have enabled an assessment of whether there had been change from one period to the next but not whether individual countries or the EU as a whole made progress towards an overall objective. This would have been even more difficult where the study team was only able to rely on information on new measures or initiatives rather than on more measurable actions, outputs or outcomes. A2.7.4 Reliance on measuring inputs Most of the ERA actions were being measured by inputs in the Member States and Associated Countries rather than through outputs or outcomes for priorities 1 and 2. Output and outcome indicators were available across a limited range of actions for priorities 3, 4, and 5. Progress measures would be more robust by reference to activities and results over time (input, outputs and outcomes) rather than through a focus on inputs. 2 April

66 Annex 3 Outcome of scope test A3.1 Priority 1 more effective national research systems Table A3.1 Action/ Outcome P1A1 P1A2 Priority 1 comparison of progress indicators and data sources Indicator Source Scope (In/Out) Share of national GBAORD allocated as project-based funding Share of institutional funding allocated on a competitive basis Indicator was not used in 2013 Share of institutions applying the core principles for international peer review Indicator was not used in 2013 MS / OECD RFO survey RFO survey In In In Indicator Source Change (Y/N) Share of national GBAORD allocated as project based funding Indicator was not carried forward to 2014 Share of project based research and development budget allocated through peer review Indicator was not carried forward to 2014 Share of institutional funding allocated based on institutional assessment and/or evaluation Outcome Indicator was not used in 2013 Share of top 10 per cent scientific publications MS / OECD RFO survey RFO survey N Y Y Y Y Scope (In/Out) In In In SCOPUS Y In A3.2 Priority 2 optimal transnational co-operation and competition Table A3.2 Action/ Outcome Priority 2 comparison of progress indicators and data sources Indicator 2013 Source Scope (In/Out) Indicator 2014 Source Change (Y/N) P2A1 Indicator was not used in 2013 National public funding allocated to transnationally coordinated R&D as per cent of GBAORD EUROSTAT Y In Scope (In/Out) 2 April

67 Share of national GBAORD allocated to transnationally coordinated research based on common priorities Assessment of the implementation of joint research agendas addressing grand challenges P2A2 Share of institutions applying international peer review standards Share of institutions mutually recognizing international peer review standards Assessment of the implementation of mutual recognition of evaluations that conform to international peer review standards as a basis for national funding decisions RFO survey RFO survey RFO survey RFO survey RFO survey In National public funding allocated to joint research agendas [within transnationally coordinated R&D] as per cent of GBAORD RFO survey N In In Indicator not carried to 2014 Y In In Share of funders which can base their project based research and development funding decisions on peer reviews carried out by non-national institutions Share of project based research and development budget allocated through peer review carried out by institutions outside the country RFO survey Y In RFO survey Y In In Indicator not carried to 2014 Y P2A3 Indicator was not used in 2013 Share of funder's research and development budget dedicated to joint defined research agendas with nonnational organisations P2A4 Share of budget allocated to transnational funding, specified by model: Lead-Agency, Money- Follows-Cooperation and Money- Follows-Researcher and other models Rate of financial commitments to the implementation (construction and operation) of the ESFRI Roadmap and to other global research infrastructures of pan- European RFO survey MS / ESFRI In Share of funders research and development budget allocated to transnational cooperation through schemes such as Lead-Agency, Money-Follows- Cooperation and Money-Follows- Researchers In Share of cumulated GBAORD committed to the construction and operation of the ESFRI Roadmap RFO survey Y In RFO survey N In MS / ESFRI N In 2 April

68 interest Rate of financial commitments to the implementation (construction and operation) of the ESFRI Roadmap and to other global research infrastructures of pan-european interest MS / ESFRI In Number of MS which have adopted a detailed roadmap with planned expenditure and related timing with regard to ESFRI ESFRI Y In P2A5 Share of non-national researchers (from MS, AC and Third Countries) accessing research infrastructure of European Interest MS In Share of non-national researchers using research infrastructure (separating other EU MS from non-eu countries) Outcome Indicator was not used in 2013 Share of scientific publications with authors from different countries (separating EU and non-eu countries) MS N In SCOPUS Y In A3.3 Priority 3 open labour market for researchers Table A3.3 Action/ Outcome Priority 3 comparison of progress indicators and data sources Indicator 2013 Source Scope (In/Out) P3A1 Assessment of the degree of implementation of policies and measures on open, transparent and merit-based recruitment RPO survey / Euraxess / MORE survey / MS Indicator 2014 Source Change (Y/N) Out Share of organisations which systematically advertise openly first stage researchers vacancies announcements including the job profile, skills and competencies required and eligibility criteria Share of organisations which systematically advertise openly other researchers vacancies announcements including the job profile, skills and competencies required and eligibility criteria RPO survey Y Out Scope (In/Out) RPO survey Y Out 2 April

69 Share of total vacancies published on Euraxess Jobs Portal P3A2 Assessment of the degree of implementation (including financial commitment) of policies and measures supporting an enabling framework for the implementation of the HR Strategy for Researchers" P3A3 P3A4 Assessment of the degree of implementation (including financial commitment) of policies and measures supporting an enabling framework for the implementation of the "HR Strategy for Researchers" Share of identified grants which are portable across borders Share of national grants which are accessible to non-residents Share of stakeholder organisations implementing doctoral training programmes linking public and private sectors Share of PhD candidates participating in innovative doctoral training Assessment of the degree of implementation (including financial commitment) of policies and measures RPO survey Out Share of organisations systematically publishing vacancies in Euraxess for first stage researchers Share of organisations systematically publishing vacancies in Euraxess for all other researchers RFO Survey In Share of funders supporting the uptake of Code and Charter principles in line with the HR Strategy RFO Survey In Share of institutions implementing the Charter and Code principles in line with the HR strategy where applicable RFO Survey In Share of funders whose majority of grants are portable abroad RFO Survey In Share of funders whose grants are systematically accessible to research organisations and researchers located outside the country and not belonging to intergovernmental organisations RPO survey / Euraxess Researchers' Reports Out Share of research performing organisations systematically including schemes or activities to expose PhD students to industry/other relevant employment sector RFO Survey In Share of research funding organisations systematically providing support for the implementation of structured doctoral RPO survey Y Out RPO survey Y Out RFO Survey Y In RPO survey Y Out RFO Survey RFO Survey Y Y In In RPO survey Y Out RFO Survey Y In 2 April

70 supporting structured innovative doctoral training programmes applying the "Principles for Innovative Doctoral Training" training based on the Principles for Innovative Doctoral Training P3A5 Share of research institutions implementing mobility programmes between industry and academia Outcome Share of staff participating in mobility programmes between industry and academia Share of researchers who feel that recruitment procedures are transparent, merit-based and open RPO survey Out Share of research performing organisations systematically RPO survey Out implementing programmes and/or actions to support researchers mobility outside academia MS / Euraxess Researchers' Reports In Share of researchers who feel that recruitment procedures are open, transparent and merit-based Share of non-national researchers Eurostat In Share of non-national researchers (differentiating between other EU MS from non-eu countries) Share of non-eu students in tertiary education RPO survey Y Out MORE SURVEY N In Eurostat Y In EUROSTAT Y In Share of non-eu doctoral holders EUROSTAT Y In A3.4 Priority 4 gender equality and gender mainstreaming in research Table A3.4 Action/ Outcome Priority 4 comparison of progress indicators and data sources Indicator 2013 Source Scope (In/Out) Indicator 2014 Source Change (Y/N) P4A1 Indicator was not used in 2013 Share of funders supporting systematically gender equality in research and the inclusion of gender dimension in research content RFO Survey Y In Indicator was not used in 2013 Share of research performing RPO survey Y Out Scope (In/Out) 2 April

71 organisations implementing recruitment and promotion policies for female researchers Indicator was not used in 2013 Share of research performing organisations which include the gender dimension in research content RPO survey Y Out Indicator was not used in 2013 Share of institutions which have adopted and implement Gender Equality Plans Share of funders including systematically the gender dimension in research content when allocating research and development funding RPO survey Out Share of research performing organisations which have adopted Gender Equality Plans P4A2 Indicator was not used in 2013 Share of gender-balanced recruitment committees for leading researchers in research performing organisations Outcome Indicator was not used in 2013 Share of gender-balanced research evaluation panels in research funding organisations Share of female PHD graduates, researchers, senior level in academic position and in top positions RFO Survey Y In RPO survey N Out RPO survey Y Out RFO survey Y In EUROSTAT In Share of female PHD graduates She Figures Y In Share of female researchers She Figures Y In Share of female senior researchers (grade A) Share of females who are head of organisation She Figures Y In RPO survey Y In A3.5 Priority 5 Optimal access to and circulation and transfer of scientific knowledge Table A3.5 Priority 5 comparison of progress indicators and data sources Action/ Indicator Source Scope Indicator 2014 Source Change Scope 2 April

72 Outcome (In/Out) (Y/N) (In/Out) P5A1 Assessment of the degree of development of MS strategies for realising digital ERA in identification services, provision of digital research services and human resources factors for supporting digital science (escience) approaches MS In Indicator was not carried forward to 2014 Y Indicator was not used in 2013 Indicator was not used in 2013 Share of funders funding systematically open access to publications Share of funders funding systematically open access to data Indicator was not used in 2013 Share of research performing organisations making available on-line and free of charge [publicly funded] scientific research data systematically P5A2 Indicator was not used in 2013 Share of funders supporting systematically the implementation of knowledge transfer as part of its institutional and/or project based funding Percentage of researchers in public research organisations with experience in the private sector Indicator was not used in 2013 Indicator was not used in 2013 Number of research organisations having a dedicated knowledge transfer office MORE survey RFO Survey Y In RFO Survey Y In RPO survey Y Out RFO Survey Y In Out Indicator was not carried forward to 2014 Y Share of staff whose primary occupation is in the private sector (in Full Time Equivalents) Share of research and development budget financed by private sector RPO survey Out Share of research performing organisations having or using a structure for knowledge transfer activities RPO survey Y Out RPO survey Y Out RPO survey Y Out Share of permanent staff (by RPO survey Out Share of research performing RPO survey Y Out 2 April

73 category) employed in knowledge transfer offices organisations having dedicated staff employed in knowledge transfer activities P5A3 Indicator was not used in 2013 Share of funders R&D budget dedicated to support the development and uptake of digital research services Share of MS implementing jointly developed access and usage policies for public e-infrastructures MS In Share of research performing organisations providing digital research services (i.e. cloud services, research collaboration platform, etc.) P5A4 Indicator was not used in 2013 Share of funders research and development budget dedicated to support the development and uptake of federated electronic identities Indicator was not used in 2013 Share of research performing organisations providing federated electronic identities for their researchers Rate of growth of academia held patents licensed or sold to industry Rate of growth of the number of Academia-Industry research training contracts signed RPO survey Out Rate of growth of patents held by Research Performing Organisations Rate of growth of licences held by Research Performing Organisations Rate of growth of licence income received by Research Performing Organisations RPO survey Out Rate of growth of collaborative agreements with the private sector and/or non-governmental sector Outcome Indicator was not used in 2013 Share of co-patents held with non-national institutions (differentiating between other EU MS from non-eu countries) Outcome Indicator was not used in 2013 Share of co-patents held with the private sector RFO Survey Y In RPO survey Y Out RFO Survey Y In RPO survey Y Out RPO survey Y Out RPO survey Y Out RPO survey Y Out RPO survey Y Out RPO survey Y Out RPO survey Y Out 2 April

74 A3.6 International dimension outside ERA Table A3.6 International dimension outside ERA - comparison of progress indicators and data sources Action/ Outcome Indicator 2013 Source Scope (In/Out) Indicator Source Change (Y/N) Indicator was not used in Share of organisation s research and development budget originating from third countries Indicator was not used in 2013 Share of research and development budget allocated to collaboration programmes carried out with third countries RPO survey Y Out RFO survey Y In Scope (In/Out) 2 April

75 Annex 4 Comparable indicators and 2014 Table A4.1 Indicators that remain unchanged between 2013 and 2014 Priority Action Indicator Indicator Source P1. More effective national research systems P2. Optimal transnational co-operation and competition P3. Open labour market for researchers A1. Competitive funding through calls for proposals applying the core principles of international peer review A1. Implement joint research agendas A3. Common funding principles to make national research programmes compatible, interoperable (crossborder) and simpler for researchers A4. Financial commitments for the construction and operation of ESFRI, national, regional Research infrastructures of pan-european interest A5. Access to Research Infrastructures of pan-european interest Outcome Share of national GBAORD allocated as project-based funding Share of national GBAORD allocated to transnationally coordinated research based on common priorities Share of budget allocated to transnational funding, specified by model: Lead- Agency, Money- Follows- Cooperation and Money- Follows-Researcher and other models Rate of financial commitments to the implementation (construction and operation) of the ESFRI Roadmap and to other global research infrastructures of pan- European interest Share of non-national researchers (from MS, AC and Third Countries) accessing research infrastructure of European Interest Share of researchers who feel that recruitment procedures are open, transparent and meritbased Share of national GBAORD allocated as projesct-based funding National public funding allocated to joint research agendas [within transnationally coordinated R&D] as per cent of GBAORD Share of funders research and development budget allocated to transnational cooperation through schemes such as Lead- Agency, Money-Follows- Cooperation and Money- Follows-Researchers Share of cumulated GBAORD committed to the construction and operation of the ESFRI Roadmap Share of non-national researchers using research infrastructure (separating other EU MS from non-eu countries) Share of researchers who feel that recruitment procedures are open, transparent and meritbased MS / OECD RFO survey RFO survey MS / ESFRI MS MORE Survey Note: the description of some of the indicators has changed between the two reporting periods, but the indicators are assessed as equivalent. 2 April

76 Annex 5 Indicator data availability 2013 and 2014 Table A5.1 Priority 1 - Data availability for indicators that changed between 2013 and 2014 Action/ Outcome P1A1 P1A2 Outcome Indicator Indicator Source for 2013/2014 Share of institutions applying the core principles for international peer review Share of project based research and development budget allocated through peer review Share of institutional funding allocated based on institutional assessment and/or evaluation Share of top 10 per cent scientific publications RFO survey RFO survey (similar data) RFO survey SCOPUS Table A5.2 Priority 2 - Data availability for indicators that changed between 2013 and 2014 Action/ Outcome P2A1 Indicator 2013 Indicator Source for 2013/2014 Assessment of the implementation of joint research agendas addressing grand challenges P2A2 Share of institutions applying international peer review standards Share of institutions mutually recognizing international peer review standards National public funding allocated to transnationally coordinated R&D as per cent of GBAORD Share of funders which can base their project based research and development funding decisions on peer reviews carried out by nonnational institutions Share of project based research and development budget allocated through peer review carried out by institutions outside the country P2A3 Share of funder's research and development budget dedicated to joint defined research agendas with nonnational organisations P2A4 Rate of financial commitments to the implementation (construction and operation) of the ESFRI Roadmap and to other global research infrastructures of pan-european interest EUROSTAT RFO survey RFO survey (similar data) RFO survey (similar data) RFO survey (similar data) RFO survey RFO survey (similar data) MS / ESFRI 2 April

77 Action/ Outcome Outcome Indicator 2013 Indicator Source for 2013/2014 Number of MS which have adopted a detailed roadmap with planned expenditure and related timing with regard to ESFRI Share of scientific publications with authors from different countries (separating EU and non-eu countries) MS / ESFRI SCOPUS Table A5.3 Priority 3 - Data availability for indicators that changed between 2013 and 2014 Action/ Outcome Indicator Indicator Source for 2013/2014 P3A2 Assessment of the degree of implementation (including financial commitment) of policies and measures supporting an enabling framework for the implementation of the HR Strategy for Researchers" P3A3 Share of identified grants which are portable across borders Share of national grants which are accessible to non-residents P3A4 Assessment of the degree of implementation (including financial commitment) of policies and measures supporting structured innovative doctoral training programmes applying the "Principles for Innovative Doctoral Training" Outcome Share of funders supporting the uptake of Code and Charter principles in line with the HR Strategy Share of funders whose majority of grants are portable abroad Share of funders whose grants are systematically accessible to research organisations and researchers located outside the country and not belonging to intergovernmental organisations Share of research funding organisations systematically providing support for the implementation of structured doctoral training based on the Principles for Innovative Doctoral Training Share of non-eu students in tertiary education Share of non-eu doctoral holders RFO Survey RFO Survey RFO Survey (similar data) RFO Survey (similar data) RFO Survey (similar data) RFO Survey (similar data) RFO Survey RFO Survey (similar data) EUROSTAT EUROSTAT 2 April

78 Table A5.4 Priority 4 - Data availability for indicators that changed between 2013 and 2014 Action/ Outcome Indicator Indicator Source for 2013/2014 P4A1 Share of funders supporting systematically gender equality in research and the inclusion of gender dimension in research content Outcome Share of female PHD graduates, researchers, senior level in academic position and in top positions Share of funders including systematically the gender dimension in research content when allocating research and development funding Share of female PHD graduates Share of female researchers Share of female senior researchers (grade A) RFO Survey RFO Survey (similar data) EUROSTAT EUROSTAT EUROSTAT She Figures Table A5.5 Priority 5 - Data availability for indicators that changed between 2013 and 2014 P5A1 Assessment of the degree of development of MS strategies for realising digital ERA in identification services, provision of digital research services and human resources factors for supporting digital science (escience) approaches Share of funders funding systematically open access to publications Share of funders funding systematically open access to data P5A2 Share of funders supporting systematically the implementation of knowledge transfer as part of its institutional and/or project based funding P5A3 Share of funders R&D budget dedicated to support the development and uptake of digital research services Share of MS implementing jointly developed access and usage policies for public e-infrastructures P5A4 Share of funders research and development budget dedicated to support the development and uptake of federated electronic identities MS RFO Survey (similar data) RFO Survey (similar data) RFO Survey (similar data) RFO Survey (similar data) MS RFO Survey (similar data) 2 April

79 Annex 6 Summary of evaluation progress, steps 1-3 This section summarises the evaluation status after the application of Steps 1-3 for each priority, by action and indicator. It shows where progress could be assessed based on available information and where other information would be required to fill the gaps. A6.1.1 Priority 1 more effective national research systems The first ERA priority is to establish more effective national research systems in order to increase competition and excellence within national borders and ensure the best and most effective funding allocation. The first priority consists of two actions: (P1A1) Competitive funding through calls for proposals applying international peer review; and (P1A2) Institutional funding-based on institutional assessment. The evaluation status of ERA priority 1 is summarised in Table A6.1. It shows the results from applying the tests in Steps 1-3 of the evaluation framework. Table A6.1 Priority 1 - Evaluation status Actions / Outcomes P1A1 P1A2 Outcome Reference year No. of Test 1 - indicators No. of indicators in scope Test 2 - No. of directly comparable indicators Test 3 - No. of indicators for which info can be obtained in the other reporting period Summary Two indicators could be used to assess progress Two indicators could be used to assess progress n/a n/a n/a One indicator could be used to assess progress A6.1.2 Priority 2 optimal transnational co-operation and competition The second ERA priority is to establish a common research agenda, improve interoperability and comparability of national programmes, and build effective pan-european research infrastructures. The second priority consists of five actions: (P2A1) Implement joint research agendas. (P2A2) Mutual recognition of evaluations that conform to international peer-review standards. (P2A3) Common funding principles to make national research programmes compatible, interoperable (cross-border) and simpler for researchers. The following two actions are specific to the priority area on research infrastructure : 20 (P2A4) Financial commitments for the construction and operation of ESFRI, national, regional Research infrastructures of pan-european interest; and (P2A5) Access to Research Infrastructures of pan-european interest. 20 In the first progress report, ERA actions specific to research infrastructure were part of priority two optimal transnational co-operation and competition. The updated indicator list provided by DG RTD has classified the ERA actions specific to research infrastructure under the priority research infrastructures. 2 April

80 The evaluation status of ERA Priority 1 is summarised in Table A6.2. It shows the results from applying the tests in Steps 1-3 of the evaluation framework. Table A6.2 Priority 2 - Evaluation status Actions / Outcomes P2A1 P2A2 P2A3 P2A4 P2A5 Outcome Reference year No. of Test 1 - indicators No. of indicators in scope Test 2 - No. of directly comparable indicators Test 3 - No. of indicators for which info can be obtained in the other reporting period Summary indicators could be used to assess progress indicators could be used to assess progress indicators could be used to 1 assess progress indicators could be used to assess progress indicator could be used to assess progress n/a n/a n/a indicator could be used to assess progress A6.1.3 Priority 3 open labour market for researchers The third ERA priority is to increase researcher mobility, training and attractive career development. It contains five actions: (P3A1) Open, transparent and merit based recruitment of researchers. (P3A2) Researchers careers. (P3A3) Cross-border access to and portability of national grants. (P3A4) Support structured innovative doctoral training programmes; and (P3A5) Support mobility between private and public sector. The evaluation status of ERA priority 1 is summarised in Table A6.3. It shows the results from applying the tests in Steps 1-3 of the evaluation framework. Table A6.3 Actions / Outcomes P3A1 P3A2 Priority 3 - Evaluation status Reference year No. of Test 1 - indicators No. of indicators in scope Test 2 - No. of directly comparable indicators Test 3 - No. of indicators for which info can be obtained in the other reporting period Summary No indicator data to use for assessment; other info. sources need to be consulted indicators could be used to 0 assess progress P3A indicators could be used to 2 April

81 P3A4 P3A5 Outcome assess progress indicators could be used to 0 assess progress n/a n/a n/a No indicator data to use for assessment; other info. sources need to be consulted indicators could be used to 1 assess progress A6.1.4 Priority 4 gender equality and gender mainstreaming in research The fourth ERA priority aims to make better use of diverse scientific human resources as a way to foster quality and relevance of research. The fourth priority consists of two actions: (P4A1) Foster cultural and institutional change on gender; and (P4A2) Gender balance in decision making process. The evaluation status of ERA priority 1 is summarised in Table A6.4. It shows the results from applying the tests in Steps 1-3 of the evaluation framework. Table A6.4 Priority 4 - Evaluation status Actions / Outcomes P4A1 P4A2 Outcome Reference year No. of Test 1 - indicators No. of indicators in scope Test 2 - No. of directly comparable indicators n/a n/a n/a n/a n/a Test 3 - No. of indicators for which info can be obtained in the other reporting period Summary 2 indicators could be used to assess progress 1 indicator could be used to assess progress indicators could be used to assess progress A6.1.5 Priority 5 Optimal access to and circulation and transfer of scientific knowledge The fifth ERA priority aims to improve access to and uptake of knowledge and facilitate innovation. The fifth priority consists of four actions: (P5A1) Open access for publications and data resulting from publicly funded research. (P5A2) Open innovation (OI) and knowledge transfer (KT) between public and private sectors. (P5A3) Harmonise policies for public e-infrastructures and associated digital research services; and (P5A4) Uptake of federated electronic identities. The evaluation status of ERA priority 1 is summarised in Table A6.5. It shows the results from applying the tests in Steps 1-3 of the evaluation framework. 2 April

82 Table A6.5 Actions / Outcomes P5A1 P5A2 P5A3 P5A4 Outcome Priority 5 - Evaluation status Reference year No. of No. of indicators indicators in scope No. of directly comparable indicators No of indicators for which info can be obtained in the other reporting period Summary indicators could be used to assess progress n/a n/a indicator could be used to assess progress indicators could be used to assess progress n/a n/a n/a n/a n/a n/a 1 indicator could be used to assess progress No indicator data to use for assessment; other info. sources need to be consulted A6.1.6 International Dimension outside ERA This priority takes into account the globalisation of knowledge and research and aims to build international cooperation for research. This priority has one action: Openness of Member State/Associated Country for international cooperation The evaluation status of ERA priority 1 is summarised in Table A6.6. It shows the results from applying the tests in steps 1-3 of the evaluation framework. Table A6.6 International Dimension outside ERA - Evaluation status Action / Outcome Reference year No. of No. of indicators indicators in scope n/a No. of directly comparable indicators 0 No of indicators for which info can be obtained in the other reporting period n/a Summary Baseline could be established for this outcome, but not progress 2 April

83 Annex 7 ERA intervention logics Figure A7.1 Intervention Logic ERA Priority 1 More effective national research systems ERA Priorities (Objectives) Problem areas ERA Actions Activities Outputs Outcomes Impacts 1. More effective national research systems Limited public resources for RTD Insufficient competition in national research systems Strong variation in share of competitively allocated funding across EU Little institutional funding based on performance criteria Strong overlap in research profiles of RFOs and RPOs, no specialisation. 1a. Competitive funding through calls for proposals applying international peer review 1b. Institutional funding-based on institutional assessment MS/AC and RFOs design or amend national research and innovation strategies and funding mechanisms MS/AC introduce qualitative performance goals for institutional funding mechanisms increased share of competitively allocated funding through RFO in total RTD spending. increased share of RTD budget allocated through peer review Increases share of institutional funding allocated to RPOs based on institutional assessment and/or evaluation and performance-related indicators. Increased number of high-impact scientific publications Increased social impact of research Increased number of patent applications and co-patents Improve capacity and efficiency of national research systems Higher degree of specialisation Higher performance in scientific and commercial output Less overlap in research and scientific profiles 8 May 2 April

84 Figure A7.2 Intervention Logic ERA priority 2 Optimal transnational cooperation and competition 8 May 2 April

85 Figure A7.3 Intervention Logic ERA priority 3 Open Labour Market for researchers 8 May 2 April

86 Figure A7.4 Intervention logic - ERA priority 4 Gender equality and gender mainstreaming in research 8 May 2 April

87 Figure A7.5 Intervention logic - ERA priority 5 Optimal access to and circulation and transfer of scientific knowledge 8 May 2 April

88 Figure A7.6 Intervention logic - ERA priority International dimension outside ERA 8 May 2 April

89 Annex 8 Indicator maps and indicator appraisal tables This annex provides details of the indicator maps and indicator appraisals discussed in section 3. Indicator maps and appraisals are provided together for each priority area. Indicator maps present the complete set of indicators identified for the priority area alongside the priority s actions, although indicators should not be read directly against actions. A key to the indicator maps is provided in Table A8.1. A table appraising each indicator is also provided. Table A8.2 provides the scoring system used for the assessment. This is followed by an appraisal of the indicators suggested by DG RTD. Table A8.1 Map colour / shading Yellow Blue dots Grey dots Green Indicator maps - key Indicators suggested by DG RTD for particular consideration. These are included in the wider indicator appraisal, but additional appraisals with more detailed comments are also provided for these for each ERA priority area. Indicators that have been used in ERA progress reports. Indicators that have been used in the Commission s RPO survey. Assessment of these indicators is out of scope for this study, but are presented here to demonstrate where information about research performing organisations is being used to assess ERA progress. i) Indicators which have been identified in reports prepared by expert groups reviewing the ERA monitoring mechanisms (e.g. Doussineau et al., 2013; EC, 2008b, 2009, 2013b) (ii) indicators available from datasets considered for this study (OECD, Eurostat, the Innovation Union Scoreboard, etc.); and (iii) indicators that can be derived easily from the separate activities identified through the intervention logics. These indicators could be useful to fill gaps in the monitoring framework, particularly in the measurement of outputs and outcomes. Table A8.2 Indicator scoring system Criterion Availability (frequency, timeliness) Reliability Relevance Completeness Accessibility Ability to assess the effectiveness of ERA Scoring Not available: 0 Low: 1 Medium: 2 High: 3 Not at all: 0 Low: 2 Medium: 4 High: 6 Not at all: 0 Low: 2 Medium: 4 High: 6 Not at all: 0 Low: 1 Medium: 2 High: 3 Not at all: 0 Low: 1 Medium: 2 High: 3 Not at all: 0 Low: 1 Medium: 2 High: 3 Overall score Sum of scores (out of 24) 2 April

90 A8.2 Priority 1 More effective national research systems Figure A8.1 Priority 1 potential input, output and outcome/impact indicators 8 May

91 Table A8.3 Indicators Appraisal - Priority Area 1 Indicator Type Data Source Information collected (denominator/ numerator) Share of the budget of R&D project-based funding allocated through a peer review process Share of the national institutional funding allocated on the basis of institutional assessments Input Availability (Frequency, timeliness) Reliability Relevance Completeness Accessibility Ability to assess effectiveness of ERA This indicator is n/a n/a Future ERA Project-based n/a n/a Low not reported in surveys could funding typically official STI produce data for allocated based statistics, nationally. a majority of MS, on peer review but robustness of the estimates is It has not so far uncertain: data been collected are not gathered through ERA nationally and surveys, but there are major could be included differences in in future surveys. funding systems and definition of peer review procedures. Input Data collected through the ERA Survey n/a High Data is obtained through a survey, 1-2 year s data available; issues include: (i) low The allocation of The survey institutional R&D collected data for funding based on EU MS in 2013, past performance with 21 countries is an accepted providing a MS response means by which response. rates (ii) data are to increase not formally effectiveness of reported in most research MS so survey systems. But responses are there is no estimates and evidence that a (iii) variability in greater share of the working assessmentbased funding definitions used in preparing leads to better estimates. results. High Low Score /24 8 May

92 Indicator Type Data Source Information collected (denominator/ numerator) Share of block and institutional funding allocated using performancebased criteria, as share of national GBAORD Input Data not available (Suggested by ERA expert group (2013)) Not available Institutional performance-based funding (IPBF) / GBAORD Availability (Frequency, timeliness) Reliability Relevance Completeness Accessibility Ability to assess effectiveness of ERA n/a Annual n/a Simple estimations IPBF of n/a Highly relevant n/a n/a n/a Possible for Relevant, although ERAC delegations the results / target to would need careful make a rapid calibration assessment Total GBAORD Input OECD Main GBAORD / GDP Annual, time High Medium High High Could serve as an as per cent of Science and series, oneyear input indicator for GDP Technology or two- measuring progress Indicators, year time lag of both ERA actions Eurostat within this priority (competitive funding based on calls for proposals and institutional funding based on institutional assessment) Degree to which Input ERA Survey MS with provisions High Medium High High Medium Could serve as an MS use for using the core indicator because international principles for ERA survey assesses experts in its peer international peer ERA progress. review review / all MS Degree to which Input ERA Survey RFOs using High Medium High High Medium Could serve as an MS use appropriateness and indicator because evaluation criteria excellence as ERA survey assesses for allocating evaluation criteria in ERA progress. Score /24 8 May

93 Indicator Type Data Source Information collected (denominator/ numerator) project funding based Degree to which MS institutional funding is linked to performance Share of the National GBOARD allocated through project-based funding (as opposed to institutional funding) Share of Output competitive funding vs. block funding as share of national GBAORD the peer review / all RFOs Input ERA Survey MS with institutional funding linked to performance / All MS Output Data collected through the ERA Survey 2014 Eurostat (gba_fundmod)) Institutional [block] funding in MEURO / total R&D appropriations in MEURO Availability (Frequency, timeliness) Reliability Relevance Completeness Accessibility Ability to assess effectiveness of ERA High Medium High High Medium Could serve as an indicator because ERA survey assesses ERA progress Data formally recorded in a small number of EU MS, so based largely on estimates The survey collected data for the EU MS in 2013, with 21 countries providing a response Project-based funding is an accepted means by which to generate competition among researchers and improve quality and productivity overall present Medium High Low High Lack of evidencebased target: ability to measure effectiveness is therefore low; in combination with other indicators medium to high Score /24 atent applications in grand challenges per billion GDP (in Outcome/ Impact OECD Total patents (granted, EPO) / patents granted by grand challenge Only total patents available; disaggregated High Medium (environmental patents), patents <2000 all High only for environmental patents. For other grand challenges Low effectiveness. Focus is on one or two grand challenges; innovation output 8 May

94 Indicator Type Data Source Information collected (denominator/ numerator) Availability (Frequency, timeliness) PPS Euro) data not available Number of patents per total public R&D expenditure Outcome/ Impact Eurostat Number of patents (applications at EPO) / total public R&D expenditure Reliability Relevance Completeness Accessibility Ability to assess effectiveness of ERA patents may not be a useful output measure mainly from the private sector and not on ERA <2000 High High High High Measures efficiency of public spending for innovation output. High ability to measure effectiveness Score / Trademarks per cent GDP as Revealed technological advantage in bioand nanotechnology, ICT and environment Outcome/ Impact Outcome/ Impact OECD Science Index and Technology performance Outlook (number trademark applications OECD Science and Technology Outlook of of per billion US dollars / GDP at purchasing power parity (PPP)) relative to the median in the OECD Country s share of patents in a particular technology field / country s share in all patent fields. Available, delay in the data on trademarks (taken from World Intellectual Property Organisation (WIPO)) Medium High Medium High Technology balance of payments measures international technology transfers: licence fees, patents, purchases and royalties paid, knowhow, research and technical assistance. Related to GDP, it can help to measure the effectiveness Available annually High Medium Medium High Provides an indication of the relative specialisation of a given country in selected technological domains and is based 8 May

95 Indicator Type Data Source Information collected (denominator/ numerator) The index is equal to zero when the country holds no patent in a given sector; is equal to 1 when the country s share in the sector equals its share in all fields (no specialisation); and above 1 when a positive specialisation is observed. Only economies with more than 500 patents over the period reviewed are included. Availability (Frequency, timeliness) Reliability Relevance Completeness Accessibility Ability to assess effectiveness of ERA on patent applications filed under the Patent Cooperation Treaty. Can help to measure effectiveness Score /24 8 May

96 Table A8.4 Priority 1 - appraisal of indicators suggested by DG RTD Proposed Indicator Data source / Availability Relevance Robustness Conclusion Share of the National GBOARD allocated through projectbased funding (as opposed to institutional funding) Share of the budget of R&D project-based funding allocated through a peer review process Share of the national institutional funding allocated on the basis of institutional assessments This indicator is not reported in official STI statistics, nationally. The data have been collected through the ERA Survey The survey collected data for the EU MS for 2013, with 21 countries providing a response This indicator is not reported in official STI statistics, nationally. Not collected through ERA surveys, but could be included in future surveys. This indicator is not reported in official STI statistics, nationally. The data have been collected through the ERA Survey The survey collected data for the EU MS for 2013, with 21 countries providing a response. Project-based funding is an accepted means by which to generate competition among researchers and improve quality and productivity overall Project-based funding allocated following the use of peer review, risk that it will be 100 per cent for everyone. Some countries / funders may still rely on high-level committees to determine winners and losers, without recourse to formal peer review. Adds little value over the metric per cent of project based funding. Support to private R&D is generally not awarded on a peer review basis in the strict sense of the term. The allocation of institutional funding for R&D based on past performance is an accepted means by which to increase effectiveness of research systems. Institutional funding is a more powerful form of intervention in a national research system, as compared with project-based funding but it is more contentious as a result. There is a high degree of variability in national funding systems, which makes this indicator less relevant. If it is taken forward, the Commission will need consider calibration of targets and international norms. Survey data collected over 1-2 years. Data not formally recorded in many EU MS, so based on estimates Not yet collected. Additional question in future ERA surveys could produce data for a majority of EU MS, but the robustness of the estimates is uncertain because data are not being gathered and reported nationally and there are major differences in funding systems and definition of peer review procedures. Data obtained through a survey, with data for 1-2 years and with some shortcomings in terms of: (i) MS response rates (ii) data are not formally reported in most MS so survey responses are estimates and (iii) variability in the working definitions used in preparing estimates. Recommended Not recommended Not recommended 8 May

97 A8.3 Priority 2 Optimal transnational cooperation and competition Figure A8.2 Priority 2 potential input, output and outcome/impact indicators 8 May

98 Table A8.5 Indicators Appraisal - Priority Area 2 Indicator Type Data Source Information collected (denominator/ numerator) Availability (Frequency, timeliness) Reliability Relevance Completeness Accessibility Ability to assess effectiveness of ERA Score / 24 Share of national public funding for R&D transnationally coordinated, expressed as a percentage of GBOARD Input Not collected n/a n/a The share of national funding where this condition of transnational coordination applies would necessarily be small, and subject to so many external factors as to be meaningless The involvement of non-national research agencies / partners in the framing of national research priorities provides a useful indication of the extent to which a country is outward looking and receptive to international perspectives and priorities n/a n/a Share of the budget allocated through a peer review conducted by foreign institutions Input Not collected n/a n/a Low Relevance is unclear; risk of only identifying structural differences between smaller and larger research systems n/a n/a Low May

99 Share of the national GBAORD invested in the construction and operation of research infrastructures listed on the ESFRI roadmap Input ERA Survey High Medium Good example of EU added value and a relevant indicator: EU has experience developing international research infrastructure Data collected through the ERA Survey, but only three MS provided estimates High High Degree to which MS participate in ESFRI Input ERA Survey MS participating in the development of at least one of the research infrastructures identified by ESFRI / all MS High Medium High High Medium Could serve as an indicator because ERA survey assesses ERA progress Degree to which MS engage in transnational cooperation via an EU framework programme. Input ERA Survey RFOs implementing cooperation activities without EU framework programmes / all RFOs High Medium High High Medium Could serve as an indicator because ERA survey assesses ERA progress Amount and share of joint Output Not collected Numerator: Total Data research agendas amount (in ) of joint collected initiatives addressing research agendas grand challenges, which addressing grand are subject to ex post evaluation, ERA expert challenges in MS. Denominator: Total group (2013) amount (in ) of joint research agendas in MS. not n/a High Indicator provides information on joint research agendas which shape future research. n/a no dataset available Low - No dataset available but data collection possible MS via The indicator may identify the share of joint research on grand challenges, but this requires a measureable definition of grand challenge May

100 Share of public funding allocated to transnational R&D cooperation, ERA expert group (2013) Output Eurostat Numerator: public funding allocated to transnational R&D cooperation Denominator: total public funding on R&D by MS High High - Indicator provides information on share of transnational R&D funding High High High It indicates a growth or decline concerning international cooperation Share of national GBAORD allocated to transnationally coordinated research based on grand challenges, ERA expert group (2013) Output Not collected Numerator: national GBAORD allocated to transnationally coordinated research based on grand challenges Denominator: total national GBAORD of MS n/a Breakdown by grand challenges not available n/a High - Indicator n/a No provides information dataset on available share of transnational R&D funding on grand challenges Low - No dataset available but data collection possible MS via Medium High. Variant of previous indicator. Breakdown by grand challenges may not be necessary for measuring ERA effectiveness Financial commitments to Output research infrastructures, categorised as approved, under review and possible and by date for expected decisions regarding future funding, ERA expert group (2013) Data collection possible MS Numerator: Total national funding via earmarked for research infrastructures Denominator: Total funding on national infrastructures n/a - data has not been collected yet n/a yet Medium The indicator provides information on financial commitments for research infrastructure as per cent of total expenditure on infrastructures n/a no dataset available Low - No dataset available but data collection possible via MS. Proxy could be developed based on Structural Funds (SF) appropriations. Medium The indicator is based on the national definition of research infrastructure; lacks a common definition. Poor indicator of ERA effectiveness given large discrepancies of scale and scope of existing research infrastructures. Requires demand 8 May

101 analysis relevance. for List of national actions designed to enhance cross-border access of scientific researchers, ERA expert group (2013) Output Data collection possible MS via List of actions and regulations published on government research portal n/a - Data has not been collected yet n/a yet Low Indicator is unclear on what will be counted. Counting does not indicate intensities. n/a No dataset available Low - No dataset available but data collection possible MS via Low - indicator based on counting measures and not on intensities Per cent of research performers experiencing problems accessing to research infrastructure Output ERA Survey Research performers experiencing problems / all research performers High Medium High High Medium Could serve as an indicator because ERA survey assesses ERA progress Number of preparatoryphase ESFRI projects in which each MS is involved, ERA expert group (2013) Outcome/ Impact Data collection possible through ESFRI projects and MS Numerator: number of preparatory-phase ESFRI projects in which MS are involved Denominator: Total number of transnational infrastructure projects in which MS are involved See Roadmap (update) ESFRI High Medium The High Data is indicator is a collected via variant on ESFRI research infrastructure / ESFRI funding High Data is available through ESFRI High Indicates effectiveness of ERA by better allocating and sharing resources Number of Outcome/ Numerator: Number of See ESFRI High Medium The High Data is High Data is High Indicates 8 May

102 implementation phase ESFRI projects in which each MS is a partner, ERA expert group (2013) Impact implementation phase ESFRI projects in which each MS is involved. Denominator: Total number of transnational infrastructure projects in which MS are involved Roadmap (update) indicator is a collected variant on ESFRI research infrastructure / ESFRI funding via available through ESFRI effectiveness of ERA by better allocating and sharing resources Impact of internationally mobile scientists, inflows versus outflows Outcome/ Impact OECD Science, Technology and Industry Scoreboard The scientific impact of researchers moving across countries is measured by proxy through the quality of the journals they publish in. Sourcenormalised impact per paper (SNIP) is the ratio of a journal s average citation count per paper and the citation potential of its subject field. 21 An impact value higher than one meansthat the medianattributed SNIP for authors of that country/category is above average. Available annually High Medium High High Indicator is based on citation impact and changes in the affiliation of scientific authors. Given its international dimension, it can serve very well for this ERA priority The citation potential represents the likelihood of being cited for documents in a particular field. Impact is estimated by calculating, for each author and mobility profile, the median across the relevant journals SNIP, over the entire period. 8 May

103 International collaboration in science and innovation Outcome/ Impact OECD Science, Technology and Industry Scoreboard International coauthorship of scientific publications is based on the share of articles featuring authors affiliated with foreign institutions in total articles produced by domestic institutions. Co-inventions are measured as the share of patent applications with at least one co-inventor located abroad in total patents invented domestically Available annually High High High High Indicator is based on co-authorship and co-invention as a percentage of scientific publications and Patent Cooperation Treaty (PCT) applications. A good indicator of this priority area given its international dimension Cross-border of patents ownership Outcome/ Impact OECD Science, Technology and Industry Scoreboard Foreign ownership of domestic inventions is measured as the share of patents invented in one country that is owned by residents in another country of total patents domestically. invented Available annually High Medium High High A good indicator of this priority area given its international dimension. Domestic ownership of inventions from abroad is measured as the share of patents owned by country residents with at least one foreign inventor of total patents owned by country residents. 8 May

104 License and patent revenues from abroad as per cent of GDP Outcome/ Impact Eurostat Numerator: License and patent revenues from abroad Denominator: GDP of MS Annual Database of policy measures for Innovation Union Scoreboard available since 2009 Medium Medium The indicator measures one component of the impact of R&D i.e. revenues from intellectual property rights (IPR) Data available for all MS Datasets are available in accessible format Medium The indicator provides important information on license and patent revenues from abroad which point to the competitive value of R&D Technology balance of payments as per cent GDP Outcome/ Impact OECD Science, Technology and Industry Scoreboard (Technology exports Technology imports) / GDP Available annually High High High High Could serve as an indicator for this priority because it measures another dimension of internationalisation in R&D International technology flows of royalties and licence fees Outcome/ Impact OECD Science, Technology and Industry Scoreboard Annual growth rate Available annually High Medium High High Could serve as an indicator for this priority because it measures another dimension of internationalisation in R&D May

105 Table A8.6 Priority 2 - appraisal of indicators suggested by DG RTD Proposed Indicator Share of national public funding for R&D transnationally co-ordinated, expressed as a percentage of the GBOARD 22 Share of the budget allocated through a peer review conducted foreign institutions by Data source / Availability Relevance Robustness Conclusion This indicator is not reported in official STI statistics, nationally. The data have not been collected previously through the ERA Survey, but several international research cooperation indicators have been reported and as such this metric could be collected / reported based on i. Share of funder's R&D budget dedicated to jointly defined research agendas with non-national funders (other EU), 2013 [RFO survey] ii. Share of R&D budget allocated to collaboration programmes carried out with third countries, This indicator is not reported in official STI statistics, nationally. The data have been collected through the ERA Survey (see Graph 11: Share of funders which can base their project based R&D decisions on peer reviews carried out by non-national funders, 2013, ERA Facts and Figures 2014) The question could be posed in future ERA Surveys The involvement of non-national research agencies / partners in the framing of national research priorities provides a useful indication of the extent to which a country is outward looking and receptive to international perspectives and priorities. It may not be a great indicator however as very few national bodies would be allowed legally to cede authority over priority setting for national funds to nonnational agencies. This type of input could be tackled through consultations and advisory mechanisms. The share of national funding where this condition of transnational coordination applies is likely to be small, and subject to many external factors. Relevance is unclear and there is a risk that the data would only reveal structural differences between smaller and larger research systems. Smaller countries may make greater use of international experts in peer review processes, for projects, institutions and disciplines. Larger countries by definition have a larger number of resident experts of international standing, and will tend to make much less use of non-residents or nonnational institutions in any peer review process. While peer reviews may be being carried out by non- Not collected. Additional question in future ERA RFO surveys could provide data, but the concept requires further definition. A survey question may produce very few responses (due to its difficulty to respond to) as seen with questions used in the ERA 2014 Facts and Figures report. Reliability and comparability of the estimates are also uncertain. Not collected. The question in the ERA Survey could be modified. But the existing ERA Survey question relates to the possibility of using non-national / non-resident institutions to inform national funding decisions on R&D projects. The response rate would likely fall dramatically if the question were brought in line with the proposed indicator (share of all national funding determined by nonnational peer review). Not recommended Not recommended 22 N.B: The final selection of indicators includes the share of funding allocated for transnational R&D. Initially similar, the critical difference is the issue of allocation: this can be measured and disclosed un-problematically, whereas the overall share of public funding that is in fact coordinated irrespective of whether the funding was explicitly put in place for this purpose opens up many of the issues highlighted above. 8 May

106 Share of the national GBAORD invested in the construction and operation of research infrastructures listed on the ESFRI roadmap This indicator is not reported in official STI statistics, nationally. The data have been collected through the ERA Survey, but only three MS provided estimates. residents / non-nationals, their advice is not used to define national budgets A good example of EU added value and a relevant indicator. The EU has experience of developing international research infrastructure. Data variability (due to contextual and structural differences) would be problematic. EU MS would need to commit to providing this information. Not recommended 8 May

107 A8.4 Priority 3 Open labour market for researchers Figure A8.3 Priority 3 potential input, output and outcome/impact indicators 2 April

108 Table A8.7 Indicators Appraisal - Priority Area 3 Indicator Type Data Source Information collected (denominator/ numerator) Availability (Frequency, timeliness) Reliability Relevance Completeness Accessibility Ability to assess effectiveness of ERA Score /24 Share of MS that have provided guidance material on open, transparent and merit-based (OTM) recruitment Input ERA Progress Report (2014) Low Medium Low Low Results are based on low response rate, thus low reliability of the indicator Research funding available for mobility scholarships and stipends as a proportion of the total funding for research Input MORE2 Total available for mobility of researchers by institution Annual financial plan High High High High / Medium High Funding creates opportunities for mobility Number of dual and joint degrees as of total of degrees by MS Input MORE2, JRC Number of joint degrees by MS. The number of dual and joint degrees divided by total. Annual programme Medium Medium Medium Medium High dual and joint programmes provide opportunities for mobility Share of job offers within the national public research system published on EURAXESS or equivalent websites Output Not collected (proxy available and approved by WG Monitoring of ERA SGHRM, published in Researchers' n/a Open publication of appointments is not equivalent to appointments being decided based on merit It also only relates to the public sector, where the This indicator is n/a n/a n/a relevant to the ERA objective, and would be a useful addition to the portfolio of metrics 2 April

109 Reports) ERA objectives are multi-sectoral Share of organisations with EC HR Excellence in Research Acknowledgement Output Data available from the Commission website (also reported in JRC 2013) Shares are not calculated, although they could be. Periodically updated High High High High High Joint research projects or publications as a proportion of the total number of projects or publications Output Data not available, indicator suggested in JRC Number (per cent) of joint research projects. Dividing the number of joint research projects by the total number of the research projects Not available n/a High n/a n/a Medium The indicator is relevant but has to be assessed against other data Researcher posts advertised through EURAXESS Output EURAXESS Updated frequently, statistics available from Commission Services High Low High Given that no qualitative assessment of the job vacancies is provided and the focus is purely on numbers, the indicator s ability to assess ERA effectiveness is low Non-EU doctorate holders as Output Innovation Denominator: total Low Two Medium Medium Low - only part High Eurostat Latest dataset 2 April

110 a per cent of total doctorate holders Scoreboard - Eurostat doctorate holders in MS Numerator: non-eu doctorate holders in MS Eurostat samples available (2006 and 2009) of EU-28 MS covered in Eurostat provides several data formats to access primary and aggregated data Proportion of doctoral candidates with citizenship of another EU MS Outcome/ Impact IU Scoreboard indicator Proportion of doctoral candidates in each MS with citizenship of another EU MS High Medium High DG RTD s annual Researchers Report has been carried out annually for several years, and this question has a high response rate and links to what appear to be robust national statistics High High Average amount of time spent outside of academia during PhD studies Outcome/ Impact MORE2 survey Non-regular survey High High High Limited Share of researchers that have worked abroad Outcome/ Impact MORE2 Non-regular survey High High High Limited Non-EU PhD students as a per cent of total PhD students Outcome/ Impact Eurostat Number (per cent) of international Annual data High Medium High Medium Medium The indicator will only 2 April

111 PhDs. Dividing the number of international PhDs by the total PhDs provide data about non-eu mobility. Mobility within EU would be excluded Researchers per thousand labour force, new doctoral graduates overall and in S&E per thousand labour force Outcome/ Impact Eurostat <2000 High High High High Limited for assessing ERA, focus on innovation and business sector Researchers working in the business sector Outcome/ Impact Eurostat <2000 High High High High Share of researchers who feel that mobility had positive impacts on qualifications Outcome/ Impact MORE2 Non-regular survey High High High Limited April

112 Table A8.8 Priority 3 - appraisal of indicators suggested by DG RTD Proposed Indicator Share of job offers within the national public research system published on EURAXESS or equivalent websites Proportion of doctoral candidates with citizenship of another EU MS Scoreboard indicator) (IU Data source / Availability Relevance Robustness Conclusion This specific indicator is not reported in official STI statistics, nationally. The data have not been collected through the ERA Survey. Euraxess has data on researcher recruitment, which could be used to profile usage by country and by field. Some EU MS with highly centralised researcher HR functions may have data on national recruitment / appointment numbers. These data are not available for most EU MS, and would need to be obtained through surveys of RPOs. MORE2 has done similar work, so it would be feasible albeit costly. This indicator is not reported in official STI statistics, however OECD MS do report on international student numbers within tertiary education nationally, therefore the proportion of research students in a country can be compared with other countries. The indicator is reported in the IU Scoreboard, and is taken from DG RTD s annual Researchers Report. This indicator is relevant to the ERA objective, and would be a useful addition to the portfolio of metrics. But open publication of appointments is not equivalent to appointments being decided based on merit; it only relates to the public sector, where the ERA objectives are multisectoral. The MORE2 survey took a different approach, asking researchers whether they judge recruitment to be OTM (e.g. share of university-based researchers satisfied with the extent to which research job vacancies are publicly advertised and made known by their institution, Europe, 2012). This kind of question is problematic inasmuch as individual job candidates have a necessarily narrow view of a potential employer s HR procedures. Nonetheless, this kind of partiality applies across MS and so while the results will be biased they are likely to be equally biased in all EU MS. The share of non-eu doctoral candidates as a percentage of all doctoral candidates is a useful indicator of the openness and attractiveness of a research system. The focus on international students from other EU MS is of special interest from the perspective of ERA, but it excludes non-eu countries, including countries that are driving growth in international student mobility (China, India, Latin America, etc.) and well-established sources of students from Japan and the US. The data have not been collected through the ERA Survey. An indicator could be created using Euraxess data, however, that may be too partial to be useful; many employers publish job opportunities on other open web sites, at which point the indicator may only reveal the extent to which Euraxess is a preferred communication channel for researcher employers in different countries / disciplines). Re-running an RPO survey would be more robust, but would be rather costly and especially so if one wants to capture both the public and private sectors DG RTD s annual Researchers Report has been carried out for several years, and this question has a high response rate and links back to what appear to be robust national statistics. Not recommended Recommended 2 April

113 A8.5 Priority 4 Gender equality and gender mainstreaming in research Figure A8.4 Priority 4 potential input, output and outcome/impact indicators 2 April

114 Table A8.9 Indicators Appraisal - Priority Area 4 Indicator Type Data Source Information collected (denominator/ numerator) Availability (Frequency, timeliness) Reliability Relevance Completeness Accessibility Ability to assess effectiveness of ERA Score /24 Share of national RPOs which have adopted a gender equality plan Input ERA Survey Share of national RPOs which have adopted a gender equality plan High The presence of An input a gender indicator that equality plan highlights a first does not indicate step that whether the plan institutions might is enforced. take towards achieving greater gender equality. The ERA Survey collected data for The low response rate to this question decreases the robustness of this indicator. High Medium Proportion of funding allocated to projects that integrate gender aspects in science and technology research (also known as gender mainstreaming in science / gender dimension in research content) Input Not available n/a n/a Reliability uncertain, given that integration of gender aspects might take many different forms Potentially valuable indicator gender mainstreaming of n/a Data not collected High gender dimension in research gives insight to awareness of the researchers and the advances of the field with regards to gender Number of applicants and beneficiaries of research funding by sex Input She Figures, based on WiS database (DG Research and Innovation). Number male/female applicants of Annual data High High Data not available for all MS She Figures / High Indicator WiS database suggests the (DG RTD). degree to which there is a gender imbalance in 2 April

115 research funding applications, but also reflects the gender balance in research positions Compound annual growth rate of PhD (ISCED 6) graduates, by sex Input She Figures (2013) based on Eurostat - Education Statistics (online data code: educ_grad5); IT - MIUR- Italian Ministry of Education ( ). Percentage change Annual data High High The higher the number of graduates the higher the probability of more female researchers on the labour market Data available for all MS Data are accessible and available through Eurostat and She Figures Medium the indicator has assessed in relation to other indicators to give a complete picture of ERA priorities (e.g. actual employment of female researchers after graduation) Proportion of female PhD (ISCED 6) graduates Input She Figures Female graduates / (2013) based male graduates on Eurostat - Education Statistics (online data code: educ_grad5); IT - MIUR- Italian Ministry of Education ( ). Annual data High High The higher the number of graduates the higher the probability of more female researchers on the labour market Data available for all MS Data are accessible and available through Eurostat and Figures Medium The indicator has to be assessed in relation to other indicators to give a complete She picture of ERA priorities (e.g. actual employment of female researchers after graduation) 2 April

116 Proportion of female PhD student graduates in science and technology Input Eurostat Female graduates / male graduates Annual data High High The higher the number of graduates in science and technology, the higher the probability of more female researchers on the labour market Data available for all MS Data are accessible and available through Eurostat Medium The indicator has to be assessed in relation to other indicators to give a complete picture of ERA priorities (e.g. actual employment of female researchers after graduation Share of gender-balanced research evaluation panels within funding organisations Output ERA Survey Share of genderbalanced research evaluation panels within funding organisations High Whilst greater Introducing the Current data High participation of gender covers funders women in dimensions into who answered research the research the ERA survey evaluation panels funding process in 2014, which is is a core aspect represent 34 per desirable, this of gender cent of total EU indicator would mainstreaming GBAORD. not provide A higher information on response rate seniority in these would make this panels. Existing dataset more inequalities of robust. status in the research sector may easily be reproduced. Medium April

117 Proportion of female academic staff Output She Figures, Headcount of female based on data staff as a proportion from WiS of male staff database (DG Research and Innovation). Annual data High High Data missing for some MS She Figures, High Indicator WiS database reflects gender (DG Research balance in the and Innovation). academic workforce Share of female researchers on temporary contracts vs. non-temporary contracts across career paths, JRC Synthesis report (2013) Output Eurostat. Further data can be collected through MS Composite indicator with partial data available. Eurostat - Share of women researchers (FTE) for all sectors. There are no data on type of contracts. Eurostat data available annually Medium Medium - This indicator highlights employment permanence and security rather than seniority, an additional element of gender imbalance in research. Eurostat data is available for all MS. Eurostat data are accessible and available through an online platform that can be further analysed. Medium - The indicator can provide information on temporary and permanent researchers contracts to strengthen the gender dimension in research programmes Distribution of researchers in the higher education sector, by sex and age group Output She Figures Data includes R&D expenditure and R&D personnel for the following categories: business enterprise, government, higher education, and private / non-profit. Annual data since 2003 is available on Eurostat High High - The indicator provides information on female researchers according to different age groups (<35 years, years, years, and 55+ years) Data available for all MS Data are accessible and available although Eurostat although She Figures are not compiled in a database but provided in an annual report. High - In the higher education sector, the greatest gender differences are observed in the two extreme age groups, among the youngest researchers aged under 35 and among those above 55 years of 2 April

118 age Distribution of researchers across sector, by sex Output She Figures R&D personnel data is available (full-time equivalent (FTE)), in head count (HC), as a per cent of employment and as a per cent of labour force. Annual data since 2003 is available on Eurostat High High Indicator measures female researchers across four broad sectors of activity Data available for all MS Data are accessible and available through Eurostat, High - Indicator points to uneven distribution of female researchers in the higher education sector, however She Figures are not compiled in a government, database but provided in an annual report. the business enterprise sector and the private non-profit sector Proportion of female researchers in total labour force Output She Figures Number based on Eurostat - Human Resources in Science and Technology (online data code: hrst_st_ncat). of male/female scientists and engineers in the total labour force Annual data since 2004 High High Indicator measures gender differences in the field of science and engineering Data available for all MS Eurostat / She Figures High - Degree of gender balance in the distribution of researchers in the workforce Proportion of women employed in knowledgeintensive activities Output She Figures based on Eurostat - High-tech industry and Percentage male/female employed in knowledge-intensive Annual data since 2003 High Medium Indicator measures the presence of women in Data available for all MS Eurostat / She Figures Medium - An activity is classified as knowledgeintensive if 2 April

119 sectors knowledgeintensive services (online data code: htec_kia_emp 2). knowledgeintensive activities tertiary-educated persons employed represent more than 33 per cent of the total employment in that activity. Women score higher than men on this indicator Proportion of women in Grade A (professor) positions Outcome/ Impact She Figures/ also reported in ERA Survey The ERA survey collects data on the share of RPOs whose heads are women. She Figures captures the proportion of women in grade A (professor) positions. High High This is a highly The She Figures relevant data are indicator. collected regularly and Besides the have overall more general been identified aspect of overall as robust. female representation in research, the proportion of high-level positions additionally gives insight into the extent of a glass ceiling. High High Proportion of female grade A staff by main field of science Outcome/ Impact She Figures, based on data from WiS database (DG Headcount of female staff as a proportion of male staff Annual data High High Data missing for some MS She Figures, WiS database (DG RTD). High Indicator gender and reflects balance career 2 April

120 RTD). progression in the academic workforce Proportion of female heads of institutions in the higher education sector Outcome/ Impact She Figures, based on data from WiS database (DG RTD). Headcount of female heads of institutions as a proportion of male heads of institutions Annual data High High Data missing for some MS She Figures, High Indicator WiS database reflects gender (DG RTD) balance and career progression in the academic workforce Proportion of women on boards Outcome/ Impact She Figures, based on data from WiS database (DG RTD). Headcount of female board members as a proportion of male board members Annual data High High Data missing for some MS She Figures, WiS database (DG Research and Innovation). High - Indicator for gender balance in leadership Glass Ceiling Index Outcome/ Impact She Figures, based on data from WiS database (DG RTD). Headcount of female staff as a proportion of male staff Annual data High High Data missing for some MS She Figures, High - Indicator WiS database reflects the (DG RTD). combined effect of gender balance in student enrolment and the degree to which this population is has access to equal opportunities throughout their career progression 2 April

121 Gender pay gap statistics Outcome/ Impact Eurostat, tsdsc340 Per cent difference between average gross hourly earnings of male and female employees, as per cent of male gross earnings, unadjusted form Annual data High High Data available for all MS Eurostat Low In all sectors gender equal pay is a priority. The unadjusted Gender Pay Gap (GPG) represents the difference between average gross hourly earnings of male paid employees in enterprise and of female paid employees as a percentage of average gross hourly earnings of male paid employees The proportion of men/women researchers with children Outcome/ Impact She Figures, Female researchers based on as a proportion of Computations male researchers by the University of Brussels, Department of Applied Economics (ULB / DULBEA), based on 2010 SILC Annual data High Medium Data available for all MS She Figures, aggregate figures / EU- SILC micro data Work/life balance issues are of particular concern to researchers and their career progression 2 April

122 data April

123 Table A8.10 Priority 4 - appraisal of indicators suggested by DG RTD Proposed Indicator Share of national RPOs which have adopted a gender equality plan Share of genderbalanced research evaluation panels within funding organisations Data source / Availability Relevance Robustness Conclusion Data have been collected through the ERA Survey The Survey collected data for Data have been collected through the ERA Survey The Survey has collected data for Share of Data are not collected. women in top positions in publicly funded RPOs The ERA survey collects data on the share of RPOs whose heads are women. She Figures captures the proportion of women in grade A (professor) positions. An input indicator that highlights a first step that institutions might take towards achieving greater gender equality. The presence of a gender equality plan does not indicate whether the plan is enforced. Highlights gender equality in a particular area of the research system, but also relates to gender mainstreaming. It is of limited use for both areas, however: whilst greater participation of women in research evaluation panels is desirable, this indicator would not provide information on seniority in these panels. Existing inequalities of status in the research sector may easily be reproduced. This is a highly relevant indicator. Besides the more general aspect of overall female representation in research, the proportion of high-level positions additionally gives a view of the extent of a glass ceiling. The low response rate to this question decreases the robustness of this indicator. Current data covers funders who answered the ERA survey in 2014, which represent 34 per cent of total EU GBAORD. A higher response rate would make this dataset more robust. Robustness varies depending on which data source is used. The She Figures data are collected regularly and have been identified as especially robust. Not recommended Not recommended Recommended 2 April

124 A8.6 Priority 5 Optimal access to and circulation and transfer of scientific knowledge Figure A8.5 Priority 5 potential input, output and outcome/impact indicators 2 April

125 Table A8.11 Indicators Appraisal - Priority Area 5 Indicator Type Data Source Information collected (denominator/ numerator) Availability (Frequency, timeliness) Reliability Relevance Completeness Accessibility Ability to assess effectiveness of ERA Score /24 Share of staff employed by public research organisations dedicated to knowledge transfer activities (modification: share of organisations that has or uses a structure for knowledge transfer activities) Input The ERA survey Share of organisations collects data on that has or uses a the share of structure for knowledge organisations that transfer activities has or uses a structure for knowledge transfer activities, but this does not include a headcount Share of research Input ERA Survey Share of research and High and development development budget budget financed by the private sector financed by the private sector High Knowledge transfer only constitutes one part of this priority area, and the potential use of this indicator is further limited by the fact that an organisation s KT activities might not solely (or even predominantly) flow through official KT channels. High High High High It is problematic to express private sector investment as a share of the overall R&D budget: reducing government investment would artificially inflate this indicator and, conversely, additional government investment in R&D would decrease it. This indicator would go some way to showing the extent of transfer between research and the private sector. The Survey collected data for 2013 High Medium 2 April

126 Share of R&D public funding involving routine open access to publications (modification: Share of funders funding open access to publications) Input ERA Survey Share of funders funding open access to publications High Medium: approach to open access (OA) funding might vary between funders A relatively new phenomenon, open access publication is an essential component of ensuring circulation, access and knowledge transfer. High High High R&D in HEI s / PRO s funded by business Output Eurostat Share of (GOVERD + HERD) financed by the business enterprise sector Latest 2012 Medium data depends on firms polled High Medium / Low Medium Depends of the firms that have participated Number and share of national research performing organisations with mandatory policies for open access to and preservation of scientific information, ERA expert group (2013) Output Data collection possible via MS Numerator: Number of RPOs in MS with mandatory OA policy and preservation of scientific information Denominator: Total number of RPOs in MS n/a - Data not collected Medium Requires common understanding of OA policies for RPOs High n/a No dataset available Medium No dataset available but data collection possible via MS No dataset available Number and share of research performing organisations with interoperable and Output Data collection possible via MS Numerator: Number of RPOs with interoperable and federated repositories n/a - Data not collected High High n/a No dataset available Medium No No dataset dataset available but data collection possible via available 2 April

127 federated repositories, ERA expert group (2013) in MS Denominator: Total number of RPOs in MS MS. Number and share of research funders and research performing organisations with obligatory data management plans, ERA expert group (2013) Output Data collection possible via MS Numerator: Number of RFOs and RPOs with obligatory data management plans in MS Denominator: Total number of RFOs and RPOs in MS n/a - Data not collected Some form of data management plan is likely to be in place in most RPOs qualitative distinctions are likely to be considerable. Medium n/a No dataset available Medium No dataset available but data collection possible via MS No dataset available Non-EU doctorate Output holders as a per cent of total doctorate holders Innovation Scoreboard - Eurostat Denominator: total doctorate holders in MS Numerator: non-eu doctorate holders in MS Low Two Eurostat samples available (2006 and 2009) Medium Medium Low - Only part of EU-28 covered in Eurostat High Eurostat provides several data formats to access primary and aggregated data Latest 2009 dataset Outcome/ Impact Public-private copublication per million of the population Innovation Denominator total Low Medium High Medium Medium Captures publicprivate Scoreboard population linkages CWTS and Numerator - Number of and active Eurostat public-private coauthored collaboration research activities. Does publications. The not capture extent definition of the private of collaborations sector excludes the or distinguish private medical and between large health sector. and small firms. Publications are assigned to the 2 April

128 country/countries in which the business companies or other private sector organisations are located Firms cooperating with HEI s Outcome/ Impact CIS, Eurostat Per cent of all firms polled Latest 2010 Medium Data depend on firms polled Medium Medium / Low Medium 22 countries included Depends of the firms that have participated Firms cooperating with PRO s Outcome/ Impact CIS, Eurostat Per cent of all firms polled Latest 2010 Medium data depends on firms polled High Medium / Low Medium 22 countries included Depends of the firms that have participated Share of open access publications compared to total output of MS, ERA expert group (2013) Outcome/ Impact National and Numerator: Number of n/a Data are High High Medium - international OA OA scientific not aggregated. Restricted to MS repositories, publication in MS OA repositories which publish national statistics Denominator: number are updated on national statistics on scientific of scientific publications a regular basis on scientific publications in MS and often publications and provide MS represented statistical on OA repositories information; national statistics on scientific output are available in most MS Medium No dataset available but data collection possible via MS No dataset available Stock of doctorate holders employed in business Outcome/ Impact OECD careers of doctorate holders (CDH), United Latest 2012 Medium Medium Low Infrequent and sometimes incomplete data Low Infrequent estimates made Poor Infrequent estimates made, limited to R&D 2 April

129 enterprises Nations Educational, Scientific Cultural Organization (UNESCO) Institute Statistics Eurostat and for and capacity of firms polled Teaching in HEI s Outcome/ performed by Impact people with their primary job outside the HEI / PRO sector FTE adjunct positions in HEIs occupied by people who have their primary job in the business enterprise sector Latest 2008 Low Out of date. Variations in framework for delivering this between countries. Medium Low Difficult to collect, infrequently collected Low Infrequently collected Poor access to data, data collected too infrequently April

130 Table A8.12 Priority 5 - appraisal of indicators suggested by DG RTD Proposed Indicator Share of R&D public funding involving routine open access to publications Share of research and development budget financed by the private sector Share of staff employed by public research organisations dedicated to knowledge transfer activities Data source / Availability Relevance Robustness Conclusion These data are not reported in official statistics. However, there are some closely related indicators in the ERA survey, most notably Share of funders funding open access to publications. Data have been collected through the ERA Survey The Survey has collected data for This data are not reported in official statistics. Could be included in future ERA surveys. The ERA survey collects data on the share of organisations that has or uses a structure for knowledge transfer activities, but this does not include a headcount. A relatively new phenomenon, open access publication is an essential component of ensuring circulation, access to information and knowledge transfer. One of the few areas that genuinely relates to all aspects of this ERA priority. Shows the extent of transfer between research and private sector. The headcount concept is problematic because institutions often pool their knowledge transfer activities or use external KT facilities in some other form. It is unclear whether the number of individuals involved in the activities is necessarily a good indicator of their effectiveness. Knowledge transfer only constitutes one part of this priority area, and the potential use of this indicator is further limited by the fact that an organisation s KT activities might not solely (or even predominantly) flow through official KT channels researchers themselves might conduct such activities themselves as well. The notion of routine is unclear, and it may be preferable to look at the funding organisations rather than the share of R&D funding it is at the level of funding organisations where policies on open access are likely to occur. It is problematic to express private sector investment as a share of the overall R&D budget: lowering government investment would artificially inflate this indicator and conversely, additional government investment in R&D would decrease it. Recommended, specifically with modification to ERA survey. Not recommended, but could be modified. Not yet collected. Not recommended. However, the ERA survey s enquiry into the proportion of institutions who either have or use KT structures resolves some of the concerns noted here. 2 April

131 A8.7 International dimension outside ERA Figure A8.6 International dimension outside ERA (Priority 6) potential input, output and outcome/impact indicators 2 April