Evaluation Handbook and. Interim Evaluation EPEC. Support to the interim evaluation of the ICT research in the 7th Framework Programme.

Transcription

1 Support to the interim evaluation of the ICT research in the 7th Framework Programme Evaluation Handbook and Guidelines for the FP7-ICT Interim Evaluation September 2009 Prepared for the European Commission Directorate General for Information Society and Media Specific contract SMART No 2007/0040 Under Framework Contract No. DG BUDG No BUDG06/PO/01/Lot 1 ABAC EPEC Contact name and address for this study: Technopolis Ltd. 3 Pavilion Buildings, Brighton BN1 1EE, UK Isabelle Collins Isabelle.collins@technopolis-group.com Tel:

2 2

3 Interim evaluation of the ICT research in the 7th Framework Programme Draft Evaluation Manual INDEX 1 INTRODUCTION Purpose of the Manual Structure THE EC EVALUATION SYSTEM Evaluation of Community Activities Evaluation in DG Information Society and Media Evaluation of the Seventh Framework Programme THE INTERIM EVALUATION OF FP7 ICT Purpose of the evaluation Overview of key issues Evaluation method Main information sources Internal External Comparison other schemes/benchmarking Indicative Timetable KEY ISSUES Research quality Progress towards programme objectives Quality of implementation BRIEFING NOTES The Structure and objectives of FP ICT in FP Overall/strategic objectives Challenge level Horizontal objectives Related objectives: Complementary Activities in FP The Ideas Programme (European Research Council) Actions under the People Programme (Marie Curie) The Capacities Programme (Research Infrastructures) Related objectives: Complementary Activities out FP The FP7 Research Instruments Co-ordination national research programmes ERANETS, the JTIs and Article METHODOLOGICAL BACKGROUND Analytical framework Methodological concepts Data and evidence requirements EPEC 3

4 6.3.1 Monitoring Data Data on the external environment Additional data Methodologies Data Collection Data Analysis Measuring progress METHODOLOGICAL ANNEX: SUMMARY OF EVIDENCE BASE

5 1 INTRODUCTION 1.1 Purpose of the Manual The purpose of this manual is to provide support and guidance for the team involved in the interim evaluation of FP7-ICT. As such it is both a planning/management document, and a source of background information. It is designed to assist the evaluation panel, and to set the framework for the technical support team and the Commission services. 1.2 Structure The manual sets out the purpose of the evaluation, the key issues to be addressed, the timetable and the information sources. It then provides more information on the background to the key issues, and a series of briefing notes on the Programme. The data sources and evaluation timetable are then set out. In annexe there is some basic background information that supplements the briefing notes. While this document in its current form is a deliverable under the technical support contract, the intention is that the roadmap should be kept updated as the evaluation progresses.

6 2 THE EC EVALUATION SYSTEM 2.1 Evaluation of Community Activities The Commission defines evaluation as a judgement of interventions according to their results, impacts and the needs they aim to satisfy. 1 It is the main tool used by the Commission to assess the extent to which EU interventions reach their policy objectives and how their performance can be improved in the future. The Commission has established a framework for evaluation of its activities in which the DGs are responsible for evaluation of their activities and the central services (currently the Secretariat General) provide support and co-ordination. Where an activity covers several DGs then there will be co-ordination between them, one DG having the lead. The purpose of evaluation in the Commission is mainly to improve preparation, implementation and performance of individual policy instruments, some but still limited, input to setting political priorities. It also has a role in the communication of the achievement of policy objectives to decision makers and stakeholders as well as the challenges in achieving them. Evaluation falls in a range of activities on the theme of Better Regulation and evidence based policymaking, including the Strategic Policy and Programming Cycle and Activity Based Management initiatives. Important elements of evaluations are therefore: Relevant coverage, focus and timing to enable them to meet strategic level needs Feedback into the Commission s planning and decision making processes Ensuring that the key messages are transparent and well communicated. The Commission has a set of evaluation standards and guiding principles. Those relating to the conduct of evaluations state that: Evaluation activities must be conducted to provide reliable, robust and complete results. 1. The evaluation must be conducted in such a way that the results are supported by evidence and rigorous analysis. 2. All actors involved in evaluation activities must comply principles and rules regarding conflict of interest. 3. Evaluators must be free to present their results out compromise or interference, although they should take account of the steering group s comments on evaluation quality and accuracy. 4. The final evaluation reports must as a minimum set out the purpose, context, objectives, questions, information sources, methods used, evidence and conclusions. 5. The quality of the evaluation must be assessed on the basis of the pre-established criteria throughout the evaluation process and the quality criteria must as a minimum relate to relevant scope, appropriate methods, reliable data, sound analysis, credible results, valuable conclusions and clarity of the deliverables 2. 1 SEC(2000)1051, Focus on Results: Strengthening Evaluation of Commission Activities 2 SEC(2007)213, Responding to Strategic Needs: Reinforcing the use of evaluation 6

7 General evaluation requirements are set out in the Financial Regulation that governs the operation of the Commission. Individual programmes then generally have their own evaluation requirements, which will be found in the relevant legal instruments governing the programme (referred to as the legal base) which may set out specific requirements such as issues to be covered and the timing of evaluations. In general, evaluations take place at a number of stages in a programme s life: Figure 1 Timing of evaluations This evaluation is an interim evaluation focusing on issues of implementation and feeding into the steering of the programme and the design of future programmes. 2.2 Evaluation in DG Information Society and Media The evaluation unit in DG Information Society and Media is responsible for the evaluation of the DG s activities covering both the research activities and the IST deployment and co-ordination activities. The management of the ICT R&D activities is subject to continuous and systematic monitoring and evaluation. Each key aspect of the programme (e.g. evaluation and selection of proposals, use of instruments, networks of collaboration between participants, etc) is also subject to periodic evaluation. Resources are given to this process, in particular for the description of the ICT R&D intervention logic and the causality links between the higher level outcome objectives and the supported activities; for the development of a coherent set of results and impact indicators; for the systematic, structured, and coherent analysis of the impact of completed actions; and for the provision of high quality evidence, which is necessary for the monitoring and evaluation tasks. Activities undertaken or supported by the evaluation unit in the field of ICT research include Evaluations, including the current evaluation both covering the activities of DG Infso and contributing to the evaluation of wider initiatives including the Research and Competitiveness Framework Programmes Monitoring of the IST research implementation including the monitoring of calls and the annual monitoring of the research projects funded Studies in support of the IST-RTD Monitoring and evaluation looking at wider economic, regulatory or social issues or, in particular, developing methodological techniques to overcome some of the recognised difficulties in the evaluation of this type of activity. International collaboration initiatives working evaluation professionals in a range of countries worldwide to develop methodological approaches and exchange experience in the area of evaluation of science, technology and innovation policies. 7

8 In the case of ICT-RTD (and also the Competitiveness and Innovation Framework Programme) the activities managed by DG Information Society and Media form part of a larger programme involving a number of DGs, and where the evaluations of the ICT-focused activities feed into a wider, overall evaluation of the programme as a whole. This has implications for timing of the evaluations, and to some extent for the questions asked and methodologies employed, since a degree of coherence is necessary to provide the overall evaluations required by the budgetary authorities. 2.3 Evaluation of the Seventh Framework Programme FP7, the current research programme, is divided into four specific programmes: Co-operation ( 32.4bn): building European leadership in ten key Themes through funding research carried out by organisations working together across European and national borders. The ICT Theme a budget if 9.1bn is the largest of these. Capacities ( 4.2bn) aimed at building the infrastructure for European researchers. Within this there is an ICT element the e-infrastructures strand which has a budget of 600m Ideas ( 7.5bn) funds the European Research Council, which funds individual researchers. People ( 4.7bn) which funds strengthening the human potential of European research, notably through the Marie Curie actions. Responsibility for the overall evaluation of FP7 lies DG Research, which is the lead DG for the Framework Programme. Evaluations of the component parts of the programme lie the various DGs and departments responsible for its implementation. The Framework Programme and its constituent parts are subject to systematic appraisal that includes prior appraisal (ex-ante evaluation or impact assessment) assessment of the needs that the programme is designed to address and the choice of implementation mechanisms, interim evaluation (together the monitoring of the implementation of the programme) and ex-post evaluation at the end of the programme. Apart from the general requirements for accountability and for sound and efficient management, there is a requirement for evaluation and reporting set out in the legal base of the Framework Programme 3. This foresees the following key evaluation milestones: Ex Ante FP7 Impact (2005). This was carried out at the level of the overall Framework Programme and published as the Commission Staff Working Paper, Annex to the proposal for the Council and European Parliament decisions on the 7th Framework Programme (EC and Euratom), Impact and Ex Ante Evaluation4. Ex-post evaluation of FP6 (2008). The ex-post evaluation of FP6 IST has been completed and the final report of the panel was published in May The interim (mid-term) evaluation of FP7 the subject of this exercise In addition, to this legal requirement, the European Court of Auditors recently carried out a study on the evaluation of the Framework Programmes 6 which put an emphasis on the need for a structured and co-ordinated approach to monitoring and evaluation of the programme. 3 Decision No 1982/2006/EC concerning the Seventh Framework Programme of the European Community for research, technological development and demonstration activities ( ) 4 SEC(2005) Information Society Research and Innovation: Evaluation of the effectiveness of the Information Society Research in the 6 th Framework Programme

9 The requirements for the interim evaluation are set out in the legal base for FP7. No later than 2010, the Commission shall carry out, the assistance of external experts, an evidence-based interim evaluation of this Framework Programme and its specific programmes building upon the ex-post evaluation of the Sixth Framework Programme. This evaluation shall cover the quality of the research activities under way, as well as the quality of implementation and management, and progress towards the objectives set 7. In addition to the interim evaluation itself, a progress report is required as soon as enough data becomes available to give initial findings on the effectiveness of the new actions and the efforts made regard to simplification. The evaluation is carried out at the programme level. The overall Programme will be evaluated the support of an independent expert group, which will undertake an Interim Evaluation. Their mandate should be adopted in autumn 2009, and the evaluation should be completed in the autumn of The key questions outlined for that evaluation panel have been set out 8 as How can the impact of FP7 and future FPs on shaping the European Research Area be improved? Are the novel measures (ERC, JTIs, Article 169, RSFF) efficient respect to reaching their intended objectives? How can the impact and added value of collaborative research that cuts across scientific disciplines, industrial sectors and policy fields be further enhanced a view to better address large societal challenges? Does FP7 play an adequate role in positioning Europe on the global map of science and technology? To what extent have simplification measures been effective? Will further steps create the desired results or do we need to consider radically new approaches? 6 Report No 9/2007 concerning "Evaluating the EU Research and Technological Development (RTD) framework programmes - could the Commission's approach be improved?" 7 Decision No 1982/2006/EC Article 7(2) 8 COM(2009) 209 final Progress Report on the Seventh Framework Programme 9

10 3 THE INTERIM EVALUATION OF FP7 ICT 3.1 Purpose of the evaluation This evaluation serves two major purposes to provide guidance and steering for the final part of FP7 in particular to assist the design of the work programme for the next period, and to provide input to the design for any successor programme to FP7, since the preparations for this will begin shortly and the final evaluation will be carried out only after the start of the new programme. To satisfy the regulatory requirements, this evaluation must cover three main issues: The quality of the research activities under way The quality of implementation and management The progress towards the objectives set There is an additional requirement to assess the effectiveness of the efforts made on simplification looking beyond the implementation at aspects of the programme design. In evaluation terms this is to look at the efficiency of the programme the extent to which the desired effects are reached at reasonable cost, and the effectiveness of the programme the extent to which the objectives set (including the quality of the research) are achieved. The other evaluation issues of: Relevance the extent to which the objectives set are pertinent to the needs, problems or issues to be addressed Utility the extent to which the effects corresponded to the problems and needs to be addressed Sustainability the extent to which the (positive) effects are likely to last after an intervention has ended Will mainly be addressed in the final evaluation. However, they will be taken into account as appropriate to assist in steering the programme. Issues relating to the impact of the programme are not addressed at this point as the work is at far too early a stage of development. The evaluation also looks at the factors that have changed significantly since the previous Framework Programme, specifically the new instruments of the Joint Technology Initiatives, and Article 169 (while this existed in FP6 there were no examples in the ICT area apart from some preparatory actions). This evaluation, as well as being of immediate importance to FP7-ICT, will contribute to the overall evaluation of the 7 th Framework Programme, which will be presented to the European Parliament and the programme committee. 3.2 Overview of key issues The interim evaluation has to concentrate on three issues. These issues are set out in the legal base of the programme where they are expressed as: The quality of the research The degree of progress towards the programme objectives The quality of the implementation of the programme. 10

11 In addition to these issues set out in the legal base, there is the issue of the simplification of procedures, which arose from a number of studies and reports, and was a specific issue highlighted in the ex-post evaluation of FP6, which called for a simplification of management and a reduction of red tape. In order to make these issues operational, a series of specific questions for the panel have been identified and have been set out in section 4, below. 3.3 Evaluation method The evaluation will be undertaken by a panel of independent experts, on the basis of evidence presented to them from a number of sources. The panel will be supported by external consultants who have been contracted by the Commission to support the evaluation through the collection, collation and analysis of data, and through support in the drafting of the report. The Commission itself will provide input to the panel through written and oral submissions, and has commissioned several studies that will provide information of use to the panel. The Expert Group is composed of members from a wide selection of research performing and funding institutions. The group will also draw on the knowledge of its members, which includes technical expertise in several of the research priorities, wide coverage of different parts of the EU, expertise in evaluation and science and technology policy, and expertise in the management and organisation of large-scale R&D activities. 3.4 Main information sources The data sources for the evaluation as identified through the analysis process fall into two main categories programme information, either collected through normal monitoring supplemented by some desk research where the information exists but is not centrally codified or analysed or through additional primary data collection. These are the internal information sources. Additional information is required for the assessment of the results of the data collection this is external data either on the context or to enable comparison other programmes at international or national level Internal The sources of data for the programme are: Programme documentation Legal bases Work programmes Guidance notes and briefings Application tools and forms ICTWEB The data collected relating to the evaluation of the proposals Statistical data on proposals (budgets applied for, identification of co-ordinators and partners by institution type and country of origin, theme/call identifier, instrument, success rates) Consolidated evaluation data by country/call identifier/instrument (number/budgets of proposals not reaching quality threshold, number/budgets of proposals passing quality threshold but not funded, relevance criterion scores) External selection monitoring reports 11

12 Contracts/management data Projects data New data collected Statistical data on projects (budgets approved by partner, level of Community contribution, identification of co-ordinators and partners by institution type and country of origin, theme/call identifier, instrument, projects approved but not implemented, projects terminated early, contract amendments, administrative costs of projects) Management information by instrument and programme area (time to contract, budget execution rate, time to payment, no/% of recovery orders issued, number of cost claims errors, project extensions/overruns/late deliverables, internal management overhead, costs of ICT, number (FTE) of Commission staff involved in programme management) Interviews Project Officers and financial administrators Project reports External review reports where relevant Project outputs (especially for CSAs) Survey of participants (mainly qualitative information) Survey of non-participants Case studies of specific projects External External data falls into two categories context indicators to enable interpretation of the results and identify external developments that may support or hinder the success of the programme, and comparative indicators to enable comparison other programmes. Context indicators at the programme level are relatively difficult to find. Two main sources of economic data are Eurostat and the OECD. Most of the OECD ICT indicators measure the information economy at national level, and still focus on measures on penetration of networks and infrastructure. Of the OECD Science and Technology Indicators, again only a few are relevant. Most of these are at national level a smaller number at sector level (as a sub-set of national data). They are collected biannually, and there is a 2008 dataset. The OECD Science, Technology and Industry (STI) Scoreboard brings together over 200 internationally comparable quality indicators to explore the progress of national innovation strategies and recent developments in science, technology and industry. In the case of Eurostat there is the same division between ICT and RTD indicators similar data collected. The advantage of the OECD is the wider coverage including Associated and competitor countries. In addition to this there is sectoral information available (at a price) from commercial sources, which is relevant to many of the sectors. (For example, Gartner publish some market analyses covering key sectors in IC)T. This information is often held in the units responsible for the management of the programme challenges. 12

13 3.4.3 Comparison other schemes/benchmarking Information on other schemes is more difficult to deal, simply because of the very large number of programmes involved. These have been collected together as part of the outputs of the CISTRANA project, providing a range of extremely useful information in terms of scale and targets of the programmes. Similarly, summary/headline information in terms of costs processes and participants is available for Eureka (ITEA2 and MEDEA+) but the results of their recent evaluations and internal management indicators are not in the public domain. However, there are some data available that may be of use. In the case of the two JTIs, the participants themselves may be able to provide comparative information since there is overlap in the participation. Benchmarking information, including comparisons between ICT programme areas and between areas in the Co-operation programme need, of course to be handled care since the dangers of comparing apples and pears remains. 3.5 Indicative Timetable The indicative timetable for the study takes into account the end dates for the report which are governed by the next users in the process. As a result of the start date, however, elements of the work have been very tightly compressed and therefore there may need to be some adjustment in the data gathering to take into account the availability of interviewees and of the panel when relevant. Date Task Subtask Responsible Status Jul-09 Appointment of panel Comm Complete Selection of Chairman Comm Complete Appointment of panel members Comm Complete Programming first meeting Comm and Complete Chairman Preparation of briefing documents Tech/Comm Complete 7 Sept-09 Evaluation Panel, Kick off meeting Aug/Sept 09 Panel Briefing Comm Complete Adoption of terms of reference Panel/Comm Complete Briefing of panel on regulatory requirements (timing etc) Comm Complete Agreement on key issues for the evaluation Panel Complete Adoption of indicative Roadmap for the evaluation Panel Complete Evidence collection 9 Oct-09 Second panel meeting Tech Compilation of available information Tech In progress Internal Tech In progress Collection of additional information (in consultation panel chair) Tech In progress Preparation of self assessments Comm In progress Outline of self-assessment exercise by DG Infso Agreement on initial analyses required Allocation of tasks between panel and support team Identification of areas where new evidence may be required Comm Panel Panel/Tech Panel 13

14 Date Task Subtask Responsible Status Presentation of JTIs, Article 169 mechanism, and einfrastructures instruments Agreement of allocation of tasks inside panel according to interest/experience Evidence collection/drafting 16 Nov-09 Third panel meeting Oct 09- Dec 09 Interviews Commission officers Analysis of databases (composition analysis etc) Final drafting and administration of questionnaire Panel (list to be agreed) Discussion of outline of final structure of report Evidence collection/drafting Surveys/focus groups Analysis of survey results 14 Dec-09 Fourth Panel meeting Compilation of synthesis of evidence Presentation of first synthesis and analysis of evidence by support team Agreement on structure of final report Agreement in principle of options and potential recommendations Input to the Work programme Evidence collection/drafting Comm to coordinate Panel Tech/Panel members? Tech Tech Panel Panel Tech Tech Tech Tech Panel Panel Panel Dec 09 Input paper for Work Programme Tech Dec 09- Jan 10 Drafting of report 22 Jan-10 Fifth Panel meeting Jan-Feb 2010 Presentation of first draft report by technical support team In depth discussion of findings and recommendations Evidence collection/drafting 12 Feb-10 Sixth Panel Meeting 26 Mar-10 Seventh Panel Meeting Revision of report in line panel comments Presentation of the final report to Panel Development of presentation materials Presentation of final report to (DG and Commissioner?) Tech together Panel Tech Panel Tech Tech Panel Panel 4 KEY ISSUES The overall objectives of the ICT R7D in the 7 th Framework Programme are set out in the cooperation programme as: 14

15 Improving the competitiveness of European industry and enabling Europe to master and shape future developments in ICT so that the demands of its society and economy are met. ICT is at the very core of the knowledge-based society. Activities will strengthen Europe s scientific and technology base and ensure its global leadership in ICT, help drive and stimulate product, service and process innovation and creativity through ICT use and ensure that ICT progress is rapidly transformed into benefits for Europe s citizens, businesses, industry and governments. These activities will also help reduce the digital divide and social exclusion. These objectives set the background for the programme activities, and also for the evaluation of the programme activities. Also set out in the legal documents are the main issues to be addressed by the evaluation. This section looks at the key issues in turn, then suggests a number of specific questions that might be addressed in order to respond to the issues. In each case the questions are the result of a process involving three elements: An analysis of the objectives of the programme A process of consultation a number of key players involved in the implementation and management of the programme A review of the data and studies that are available or could be collected during the course of the evaluation The combination of the answers to the questions will enable the panel to make judgements on the four main areas on which they are asked to focus. 4.1 Research quality In order to achieve the objectives of global leadership the programme needs to demonstrate that it has attracted high quality research teams and that the research has been funded in areas where it can assist in developing or sustaining world leadership, supporting emerging technologies and building research communities so that a critical mass of resources and knowledge can be built or supported. Of course, research quality is very difficult to assess at this stage in the process. In addition the nature of research quality depends according to the stance of the various stakeholders quality priorities from an industry perspective may be different to those of academic partners. To a large extent the quality of the science can only be judged by peers. A source of information here will be the reports on individual projects, especially the small number of very large projects. Other quality indicators would normally include such things as peer-reviewed papers, but the time taken for publication is long for peer reviewed journals, bibliometric analysis, but it follows that lack of publications will affect both this and patents. It is therefore too early to make a judgement based on these. In addition, for many of the areas of research under ICT-RTD this type of academic publication is not a high priority reliance on this as a measure therefore risks under reporting. Although the classic measures of patents and publications are not available here, there are other ways of seeking evidence that the research is valued by external bodies this may be through participation in the projects in some cases or through co-funding of research activities especially through the new instruments designed to build closer links between national and international research programmes. Measures that can be examined at this point include the extent to which the programme is attracting world-class researchers in the different areas of research, the extent to which the research funded is meeting the agreed scientific needs whether this be through supporting existing areas of research or through funding new areas where breakthroughs might be expected. 15

16 The questions the panel will address include: Quality of the research 9 (a) (b) Is FP7 ICT exploiting areas of competitive advantage and at the same time able to adapt to a changing environment and to identify and explore new opportunities? Does the programme attract the best research teams 10 in Europe? How many of these are recognised as world leaders in their domains? Indicators and Evidence 11 Is FP7 ICT exploiting areas of competitive advantage and at the same time able to adapt to a changing environment and to identify and explore new opportunities? The indicators that can be applied here, based on the information available or collectable are: Indicator Evidence of transparent consultation stakeholder communities Attractiveness of the research agenda to those active in the research areas Actual and potential comparative advantage in the fields in which ICT-RTD focuses Responses to issues such as more rapid technology cycles, open innovation, need for end-to-end solutions Opportunities for exploration of new research avenues present in work programmes Sources of information Self Pre-programme reports Impact assessment Composition analysis Interviews industrial and academic leaders Self Composition analysis Interviews industrial and academic leaders Review of public-domain market research and analyses comparison of work programme priorities sector research priorities Interviews industrial and academic leaders Participant survey Interviews participants Public domain market research and analyses Does the programme attract the best research teams 12 in Europe? How many of these are recognised as world leaders in their domains? Indicator Sources of information 9 At this stage in the programme it is only possible to examine whether the necessary conditions for high quality research have been put in place 10 Best research teams include academic leaders in the field, leading industrial participants evidence of close links between the various players 11 A summary of all the indicators and the evidence base is included in the annexes 12 Best research teams include academic leaders in the field, leading industrial participants evidence of close links between the various players 16

17 Involvement of research teams recognised as world leaders in their domains Stronger engagement of industry Extent to which European research is a significant financial or strategic element of participants research portfolios Evidence of long-term commitment to research and to partnerships funded Involvement of top scientists and engineers in advisory mechanisms Nature of external funding attracted shows respect for the research Visible involvement of most respected European researchers Involvement of major organisations likely to be involved in uptake, commercialisation or development of the results Involvement of centres of excellence from outside the EU Involvement of new innovative players Self assessment Annual reports Composition analysis Composition analysis Participant survey Interviews participants Composition analysis (FP6 and FP7) Interviews participants Review of committee memberships Participant survey Interviews participants Self assessment Bibliometric study of participant in FP6 Composition analysis Interviews industrial and academic leaders Self assessment Study on high growth SMEs Composition analysis Composition analysis Self assessment Composition analysis Self assessment Study on high growth SMEs Interviews participants 4.2 Progress towards programme objectives At this stage it is not possible to look at the impacts of the programme, given that the research projects have in most cases only just started. Nevertheless it is important that the issue of whether the programme is on the right lines to achieve its objectives is one that it is important to address. This includes the objectives at all levels from the very high level objectives to the individual challenges in the ICT programme. The identification and analysis of the programme objectives is included in section 5.1, below. The objectives can be described at several different levels, particularly Broad or High-level objectives, which relate to the policy level, which are associated the intended long term impacts of the programme. They also relate to broader issues such as economic growth, competitiveness, prosperity etc and to some of the broader societal objectives. Tactical objectives, which are more programme/project specific and correlate well the intended outcomes of the programme. Operational objectives, which correlate the outputs of the programme and are operational in nature relating to the direct outputs of the programme. The evaluation of FP6 IST identified a number of strategic issues where attention was recommended. Specifically it drew attention to the need for the programmes activities to fit in the overall innovation system, and to focus on the key issues of developing new markets and improving European infrastructures and interoperability. 17

18 Based on the analysis of the objectives, the following questions will be considered by the panel. Progress towards the objectives of the ICT Specific Programme (a) How does FP7 ICT contribute to improve the positioning of Europe on the global ICT RTD map? (b) How is the programme contributing to realise the ERA objectives 13 and its 2020 Vision? (c) How is FP7 ICT positioned in the overall European innovation system, and how is it expected to contribute to the system? Is FP7 ICT employing the right mechanisms to help translate research results into innovative products, processes and services? (d) How does the programme link other European or national initiatives addressing the "knowledge triangle" of education, research and innovation? (e) Is FP7 ICT resulting in a better support of the broader EU policy agenda, notably economic growth, sustainable development, health, and meeting the challenges of an ageing society? (f) Have the einfrastructures activities effectively contributed to optimise the use and development of the best research infrastructures in Europe? To which extent has the einfrastructures approach been expanded to more application-oriented and useroriented platforms in other sectors? Evidence and indicators Indicator Contribution to improvement of positioning of Europe on global ICT map Contribution to realisation of ERA objectives Position of FP7-ICT in overall European innovation system Mechanisms to help translate research result into innovative products, processes and services Links to other European or national initiatives Sources of information Self assessment Composition analysis Participant survey Interviews participants Review of public domain market research and analyses Composition analysis Interviews participants Review against programme logic Self assessment Studies from associated programmes (eg CIP interim evaluation) Mapping of initiatives Participant survey Interviews participants and national policymakers Other studies eg Eureka, national evaluations Self assessment Participant survey Interviews participants Self assessment 13 The ERA 2020 Vision focus on three main objectives: i) modernisation of research, education and innovation systems; ii) development of European competitiveness; iii) coordinated support to researchers and research institutions, (i.e. articulating RTD activities at various levels European, national, regional as well as from both public and private funding) source 18

19 Support to the broader EU policy agenda (economic growth, sustainable development, health, ageing society) Contribution of einfrastructures activities Participant survey Interviews participants Self assessment Composition analysis Participant survey Interviews participants Interviews national policymakers Self assessment Interviews participants Interviews national policymakers 4.3 Quality of implementation While many of the programme mechanisms remain largely unchanged since FP6, there are also some changes these can be found in the way that the existing instruments are combined to achieve the objectives and in the introduction and development of the JTIs and the Article 169 Action 14. In addition to questions as to whether these have assisted the progress towards achieving the objectives, there is also the issue of how well these instruments have been introduced and their effect on the rest of the programme. ICT is a particularly dynamic, indeed turbulent, area and the issue of whether ICT research is able to respond adequately to changes in technologies, markets and the economic environment. This means that the implementation mechanisms and the work programmes through which they are executed need to be able to respond adequately to changing requirements. In addition to its technical and scientific objectives, the Framework Programme has wider socioeconomic objectives relating to the participation of specific groups. Examples of this are SMEs, where the programme seeks to promote involvement, and organisations from the new Member States where involvement is still lower. An additional group where there are objectives to improve involvement is that of increasing the participation of women in research. A sub-section of the quality of implementation is the degree of progress made towards the simplification of the programme procedures and processes. The report of the panel on the ex-post evaluation of FP6-IST recognised that the significant changes in implementation instruments limited the potential for simplification of procedures in that generation of the programme. However, the panel made a number of recommendations on the issue that should be directly addressed in FP7. These included 15 : Developing a more trust-based approach towards participants at all stages. Requiring shorter proposals fewer details of work packages and a focus on the appropriateness of partnerships, in particular the inclusion of highly innovative participants Providing more complete and helpful feedback to proposers whose ideas are not funded Testing a new approach whereby proposals are not fully evaluated initially. All applications passing a few basic checks should be given a small amount of "seed funding" for an exploratory phase. After this, exploratory projects successful results would be selected for full project funding. Financing projects based on actual performance rather than promises and 14 Details of these can be found in the briefing notes in annex 15 Information Society Research and Innovation: Delivering Results sustained impact. Evaluation of the effectiveness of Information Society Research in the 6 th Framework Programme

20 reputation could both reduce the initial paperwork and be a viable way of attracting innovative (small) companies that would not otherwise consider applying for Community funding. Exploring expanding the two-step evaluation procedure from the Open part of the "future and emerging technology" area to other parts of the programme - prospective participants first provide a broad outline of their project idea, and only provide a more refined plan once they are selected. This may increase the workload for the Commission in the early phases, and lengthen the evaluation process, but it will significantly reduce the burden on the research community of preparing proposals. Optimising reporting, which is time-consuming and may be untimely, and allow the participants to report when there is something to report. Permitting the refocusing of the research on different priorities if this becomes necessary during implementation. Permitting more flexibility in the composition of partnerships during the project, including the possibility of changing partners if the research takes a direction that would benefit from new partners or replacement of partners. Some of these issues relate to the design of the Programme at the Framework level thus not directly in the control of DG Infso. Others relate to matters of implementation and design of the work programmes where there was more potential for direct introduction of changes. The principles behind the simplification were: 1. Flexibility - providing the necessary tools to achieve FP7 objectives efficiently; 2. Rationalisation - establishing a better balance between risks and controls, avoiding procedures, rules and requests that have no added value, and aiming for the reduction of 3. Coherence clarifying rights and obligations, ensuring consistent and user-friendly communication, matching objectives and means, and taking into account participants own practices and pre-existing rules as far as possible 16 The questions to be addressed by the panel are: Quality of implementation (a) (b) (c) (d) (e) Is the process of formulating and revising the Work Programmes able to accommodate the dynamic nature of the research priorities in ICT and of new political priorities (e.g. Recovery Package)? Is the mix of instruments used and participants involved (industry, public research, academia, SMEs) adequate to achieve the objectives pursued in the various research areas? What effect has the introduction of the JTIs and the Article 169 action and the opportunities offered by the European Research Council and RSFF had on the participation in the co-operative research activities? To what extent can changes in the pattern of participation be linked to the changes in implementation methods 17 introduced in FP7? As an example, what is the perceived impact of the changes in the funding models as compared to FP6 (particularly for SMEs)? Have sufficient efforts been made to ensure that support for SMEs and for large firms is not "compartmentalised" into different measures or tools? To what extent have the changes introduced in FP7, notably the rules for participation, 16 Simplification in the 7 th Framework Programme (COM(2005)119 Final 17 These include the changes from FP6 to FP7 in the selection processes, changes in funding levels/proportions, differences in the relative success rate of project proposals. 20

21 (f) (g) and their implementation in the ICT programme simplified the application, selection and contract management processes? What is the budgetary impact of these changes? To what extent have FP7 management requirements, such resulted in reducing costs and lowering burdens 18 of participation in the programme? Has the Commission advanced in developing a more trust-based approach towards the participants? If the Commission has not advanced sufficiently, which factors are hindering it? What further improvements of the programme implementation and simplification measures should be considered? Evidence and indicators Indicator Ability of the process of formulating and revising the Work Programmes to accommodate the dynamic nature of the research priorities in ICT and of new political priorities (e.g. Recovery Package) Added value of IPs and JTIs in terms of federating activities, links between research and innovation and responding to societal challenges Existence of support activities to ensure programme effects can be realised, such as through linking of projects, dissemination and support activities Continued development of public private partnerships through FP7 (eg new JTIs ) Perceived impact of the changes in the funding models as compared to FP6 (particularly for SMEs) Changes by any major players in their participation in co-operative research in favour of the new instruments Change in the level or participation of industry, especially SMEs, compared to previous programmes Effect of the introduction of the JTIs and the Article 169 action and the opportunities offered by the European Research Council on the participation in the co-operative research activities Efforts to ensure that support for SMEs and for large firms is not "compartmentalised" into different measures or tools Sources of information Self assessment Interviews participants Interviews EC and national policymakers Mapping of research landscape Participant survey Interviews participants Interviews policymakers Self assessment Composition analysis Commission reports New JTIs developed or progressed DG Infso monitoring reports Composition analysis Participant survey Interviews participants Interviews Commission officials Composition analysis FP6 and FP7 Composition analysis FP6 and FP7 Composition analysis ICT-RTD, ERC, JTIs Interviews participants DG Infso monitoring reports Composition analysis ICT-RTD, ERC, JTIs Interviews participants and 18 These include the costs of participation, reporting requirements, costs of programme management, costs of project management 21

22 Indicator Extent to which the rules for participation and accompanying documentation are perceived to be easier to understand following the simplification initiative Changes in the costs of the application and selection processes Balance between costs and the level of support sought Evidence that the overall application and selection process has been streamlined Quality of feedback to proposers Differences between the different types of instrument (esp eg JTIs and IPs) Extent of the adoption of the proposed simplifications in funding mechanisms Level of take-up of new possibilities Costs of participation (management/coordination/reporting) for participants Management costs for the Commission Extent of implementation of flexibility in consortia regarding membership and research focus Sources of information Commission Participant survey Analysis of programme documentation Participant survey Interviews participants Participant survey Participant survey Interviews participants DG Infso monitoring reports Interviews participants and Commission Study of unsuccessful proposals Review of sample of ESRs and evaluator guidelines Evaluation monitoring reports Analysis of implementation mechanisms Analysis of comparative mechanisms DG Infso monitoring reports Interviews Commission Participant survey Interviews participants DG Infso self assessment Interviews Commission Participant survey Interviews participants and project officers 22

23 5 BRIEFING NOTES 5.1 The Structure and objectives of FP7 The Seventh Framework Programme (FP7) bundles all research-related EU initiatives together under a common roof playing a crucial role in reaching the goals of growth, competitiveness and employment; along a new Competitiveness and Innovation Framework Programme (CIP), Education and Training programmes, and Structural and Cohesion Funds for regional convergence and competitiveness. It is also a key pillar for the European Research Area (ERA). The broad objectives of FP7 have been grouped into four categories: Cooperation, Ideas, People and Capacities. For each type of objective, there is a specific programme corresponding to the main areas of EU research policy. All specific programmes work together to promote and encourage the creation of European poles of (scientific) excellence. ICT is found in the Cooperation programme. The Cooperation programme is sub-divided into ten distinct themes. Each theme is operationally autonomous but aims to maintain coherence in the Cooperation Programme and allowing for joint activities cutting across different themes, through, for example, joint calls. The ten identified themes reflect the most important fields of knowledge and technology where research excellence is particularly important to improve Europe s ability to address its social, economic, public health, environmental and industrial challenges of the future. Their continued relevance will be guaranteed by relying on a number of sources from the research sector, including the European Technology Platforms (ETP). Important themes identified in the Strategic Research Agendas (SRAs) developed by the ETPs are therefore covered by the Cooperation programme. The indicative breakdown among programmes is as follows (in EUR million): Cooperation Health Food, agriculture and fisheries, and biotechnology Information and communication technologies Nano-sciences, nano-technologies, materials and new production technologies Energy Environment (including climate change) Transport (including aeronautics) Socio-economic sciences and the humanities 623 Space Security Ideas People Capacities Research infrastructures Research for the benefit of SMEs Regions of knowledge 126 Research potential 340 Science in society 330 Coherent development of research policies 70 Activities of international cooperation

24 Non-nuclear actions of the Joint Research Centre (JCR) TOTAL The specific programme on 'Cooperation' supports all types of research activities carried out by different research bodies in trans-national cooperation and aims to gain or consolidate leadership in key scientific and technology areas. The budget will be devoted to supporting cooperation between universities, industry, research centres and public authorities throughout the EU and beyond. The bulk of EU research funding in FP7 goes to collaborative research, the objective of establishing excellent research projects and networks able to attract researchers and investments from Europe and the wider world. This is achieved through a range of funding schemes: Collaborative projects, Networks of Excellence, Co-ordination/support actions, etc, discussed in more detail below. 5.2 ICT in FP7 Within ICT the research funding is focused on seven key Research Challenges to ensure Europe becomes a world leader in ICTs. Three Challenges aim at industrial leadership in key ICT sectors, while four are driven by socio-economic targets. Within each Challenge, the Programme will fund an array of collaborative research projects, each bringing together public and private organisations across Europe to help the EU pool its scientific, industrial, financial and human resources. The 7 challenges are: ICT Challenge 1 - Pervasive and Trusted Network and Service Infrastructures ICT Challenge 2: Cognitive Systems, Interaction, Robotics ICT Challenge 3: Components, systems and engineering ICT Challenge 4: Digital libraries and content ICT Challenge 5: Sustainable and personalised healthcare ICT Challenge 6: Mobility, environmental sustainability and energy ICT Challenge 7: Independent living and inclusion In order assess progress towards the objectives, it is necessary to identify those objectives. In the case of the Framework Programme these can be found in various documents, ranging from the very high-level Lisbon objectives to the concrete objectives set out in the work programmes. In order to simplify the analysis we can break the objectives down into a set of categories by level. The programme objectives being studied in this evaluation are specifically those related to the development of the European Research Area rather than those specifically related to individual technology areas. Those will be examined later as part of the ex-post and impact studies when there has been the opportunity for outputs to have been achieved. In introducing the logic models the fact that it is important to categorise objectives by level was set out. They can be characterised as: High-level objectives, which relate to those established at the policy level (sometimes called policy objectives). These objectives correlate well the intended long term impacts of a programme or strategy. They also often relate to broader issues such as economic growth, competitiveness, prosperity etc. 24

25 Tactical objectives, which are more programme/project specific and correlate well the outcomes of a programme or a strategy (increased audience levels, establishing confidence in Europe etc). Operational objectives, which correlate the outputs of a programme and a strategy and are, operational in nature (establishment of a website, a magazine etc) In fact there is potentially a multiplicity of levels. Depending on the situation of the evaluation, it might consider higher or lower level objectives. Clearly if looking at very specific issues, while it is important to understand how (or indeed, if) they relate to the higher levels, it is not entirely realistic to expect that they will have a direct impact on them. This study situates itself at the level of ICT R&D in the Framework Programme. The high level objectives of the ICT R&D can be traced back into the Framework Programme as set out in Figure 2 below. For ICT R&D itself there are then the objectives of the challenge areas and the technologies. Individual projects (and indeed participants) will have their own objectives not all of which will be the same as, or consistent, the programme objectives. Generally a programme level evaluation will not take the objectives of individual projects or actions into account. In this case, however, the objectives of the JTIs are important because of the scale and novelty of the instruments. Figure 2 Hierarchy of objectives linking ICT R&D to overall FP Overall/strategic objectives The strategic objectives of ICT R&D are set out in the Co-operation Programme: Improving the competitiveness of European industry and enabling Europe to master and shape future developments in ICT so that the demands of its society and economy are met. ICT is at the very core of the knowledge-based society. Activities will strengthen Europe s scientific and technology base and ensure its global leadership in ICT, help drive and stimulate product, service and process innovation and creativity through ICT use and ensure that ICT progress is rapidly transformed into benefits for Europe s citizens, businesses, 25

26 industry and governments. These activities will also help reduce the digital divide and social exclusion. To translate these into concrete actions the ICT programme is further divided into a set of challenge areas, three of which aim at industrial leadership in key ICT sectors, while four are driven by socio-economic targets Challenge level The objectives of the various challenges are set out in Annex to this manual. These are mainly scientific/economic objectives of a nature that can only effectively be evaluated once the programme is complete, or at least reaching its conclusions so that sufficient time has elapsed. This evaluation is concerned questions relating to process, and particularly the extent to which progress towards the objectives is being achieved. Figure 3 Linking high-level objectives to the ICT challenges Horizontal objectives In addition to the objectives specifically related to RTD, the Framework Programme has a number of broader, mainly socio-economic, objectives to which it also aims to contribute. These include: Contributing to sustainable development Appropriate involvement of SMEs Promoting gender mainstreaming Contributing to European cohesion policy 26

27 In some cases these objectives of inclusiveness do not sit entirely comfortably the promotion of excellence, and a balance has to be sought Related objectives: Complementary Activities in FP7 In addition to the ICT-specific objectives, there are other parts of the programme that have complementary objectives and activities related to the ICT sphere, which should also be taken into account The Ideas Programme (European Research Council) The European Research Council, under the Ideas Programme, sponsors basic (frontier) research. It differs from the other parts of the Framework Programme in that it supports investigator-driven research carried out by individual teams. A question for the ICT programme is the extent to which this is complementary to the research funded under the Future and Emerging Technologies strand. The objectives of the Ideas programme are implemented through the European Research Council, which aims to support the best of the best scientific efforts in Europe across all fields of science, scholarship and engineering promote wholly investigator-driven, or 'bottom-up' frontier research. encourage the work of the established and next generation of independent top research leaders in Europe reward innovative proposals by placing emphasis on the quality of the idea rather than the research area harness the diversity of European research talent and channel funds into the most promising or distinguished researchers raise the status and visibility of European frontier research and the very best researchers of today and tomorrow put excellence at the heart of European Research Actions under the People Programme (Marie Curie) The People programme is dedicated to stimulating researchers' career development. The 'Marie Curie Actions' aims to respond to the needs of Europe's scientific community in terms of training, mobility and career development. It offers initial training of researchers through the Marie Curies networks, individual fellowships and co-financing programmes at international, national and regional level for more experienced researchers. Industry involvement is supported through actions directed at the initial training of researchers and through actions aiming to build long term cooperation between academia, industry and SMEs. The objective is to stimulate mobility between sectors and increase knowledge sharing through joint research partnerships The Capacities Programme (Research Infrastructures) The FP7 Capacities programme aims to enhance research and innovation capacities throughout Europe and ensure their optimal use. The area of most direct importance to the ICT Programme is Research infrastructures which aims to optimise the use and development of the best research infrastructures existing in Europe, including ICT-based e-infrastructures - supporting a number of interrelated topics designed to foster the emergence of a new research environment in which virtual communities share and exploit the collective power of European scientific and engineering facilities. 19 Source: ERC Website 27

28 5.2.5 Related objectives: Complementary Activities out FP EUREKA Created as an intergovernmental Initiative in 1985, EUREKA aims to enhance European competitiveness through its support to businesses, research centres and universities who carry out pan-european projects to develop innovative products, processes and services. Eureka is often cited as a comparator for ICT-FP7 although there are many fundamental differences. There are also close links between the ICT Technology Platforms and the JTIs and Eureka. EUREKA Clusters are long-term, strategically significant industrial initiatives. They usually have a large number of participants, and aim to develop generic technologies of key importance for European competitiveness, primarily in ICT. Initiated by industry in close collaboration national funding authorities, each Cluster has a technological roadmap defining the most important strategic domains. Specific goals are achieved through individual projects. Clusters aim to exploit the technologies developed through existing national and European programmes and play an important role in defining European standards and interoperability. The key clusters of importance to FP7 ICT are: CATRENE (Cluster for Application and Technology Research in Europe on NanoElectronics) aims to arrive at Technological Leadership for a competitive European ICT industry in the area of nano-/microelectronics solutions that respond to the needs of society at large, improving the economic prosperity of Europe and reinforcing the ability of its industry to be at the forefront of the global competition. CATRENE is a four-year programme, started 01 January 2008,which may be extended by another four years and follows on from the previous EUREKA programmes JESSI, MEDEA, and MEDEA+. ITEA 2 is the follow-up of the successful ITEA programme, is a strategic pan-european programme for advanced pre-competitive R&D in software for Software-intensive Systems and Services (SiS). ITEA 2 supports projects that will give European industry a leading edge in the area of SiS (in which software represents a significant segment in system functionality, system development cost & risk, system development time). EURIPIDES is a collaborative industrial R&D programme promoting smart systems and their relevant technologies. The strategy is to promote the development of new products, processes and services using or enabling the use of integrated smart systems based on microtechnology. EURIPIDES is a merger of EURIMUS II (Micro-systems) and PIDEA+ (Packaging and interconnection). 5.3 The FP7 Research Instruments The main instruments in FP7 are: (a) Collaborative projects Collaborative projects provide support for research carried out by consortia participants from different countries, aiming at developing new knowledge, new technology, products, demonstration activities or common resources for research. Within the collaborative projects a range of sizes is supported ranging from small or mediumsized focused actions to large scale integrating projects. The two instruments here are the large Integrated Projects (IPs) and the smaller STREPs. (b) Networks of Excellence NoEs provide support for Joint Programmes of Activities implemented by a number of research organisations integrating their activities in a given field, carried out by research 28

29 teams in the framework of longer term cooperation. These actions aim to establish virtual centres of research overcoming fragmentation in the research infrastructure. (c) Coordination and support actions CSAs provide support for activities aimed at co-ordinating or supporting research activities and policies (networking, exchanges, trans-national access to research infrastructures, studies, conferences, etc.). There are two types of action the co-ordination or network actions (CAs) and the specific support actions (SAs). These actions are sometimes implemented by means other than calls for proposals. (d) Individual projects: Support for frontier research These are the activities of the European Research Council including Starting Grants for young researchers and Advanced Grants for experienced researchers. These do not fall directly in the scope of this evaluation although the issue of complementarity/overlap does need to be investigated. (e) Support for training and career development of researchers These actions are mainly implemented under the Marie Curie Actions (People Programme) and are again not the main focus of the evaluation. 5.4 Co-ordination national research programmes ERANETS, the JTIs and Article 169 One of the factors that has changed significantly since the introduction of FP7, when compared to the initial design of FP6 is the emphasis on co-ordination national research instruments. FP7 has a set of objectives designed to increase the level of co-operation between and co-ordination of research programmes carried out at national or regional level in the Member or Associated States, leading to mutual opening up of programmes and development and implementation of joint activities. This may happen through activities in the programme (some CSAs) or in some cases through specific instruments such as the ERA-NETs, Article 169 actions and Joint Technology Initiatives. ERA-NETS The objective of the ERA-NET scheme is to develop and strengthen the coordination of national and regional research programmes through two specific actions: ERA-NET actions, which provide a framework for actors implementing public research programmes to coordinate their activities, either through support for new ERA-NETs or by broadening and deepening the scope of ERA-NET actions initiated through FP6 ERA-NET Plus actions where in a limited number of cases high European added value, additional EU financial support can be provided to facilitate joint calls for proposals between national and/or regional programmes Initiatives under Article 169 of the EU treaty Here, the Community participates in programmes implemented jointly by several Member States on the basis of Article 169. This is especially relevant to European cooperation on a large scale in "variable geometry" between Member States showing overlapping interests and commitments. In the case of ICT there is an Article 169 action on Ambient Assisted Living. Public research programmes only correspond to some 50% of research activities undertaken in the European Union. 29

30 Joint Technology Initiatives (JTIs) FP7 introduced initiatives aiming at integrating private and public research efforts. This is the first time that public-private partnerships, involving industry, the research community and public authorities, were proposed at European level. They are long-term Public-Private Partnerships and will be managed in dedicated structures based on Article 171 of the EC treaty. JTIs support large-scale multinational research activities in areas of major interest to European industrial competitiveness and issues of high societal relevance. Six initiatives have been identified in the "Cooperation" Specific Programme: Innovative Medicines Initiative (IMI) Aeronautics and Air Transport (Clean Sky) Hydrogen and Fuel Cells Initiative (FCH) Global Monitoring for Environment and Security (GMES) Embedded Computing Systems (ARTEMIS) Nanoelectronics Technologies 2020 (ENIAC) Of which the last two are in the domain of ICTs. 30

31 6 METHODOLOGICAL BACKGROUND 6.1 Analytical framework In line the Commission s guidance on evaluations, the evaluation is predicated on a definition of the logic behind the intervention the programme or intervention logic. This sets out the principle that there are a set of needs to be addressed which give rise to a set of objectives. These objectives exist at various levels, ranging from high-level policy and economic objectives to extremely operational objectives. The programme then applies a set of resources to achieving these objectives through a set of activities which generate some immediate outputs. The logic then implies that these outputs in turn will lead to short-term outcomes which in time lead to some intermediate outcomes and eventually some long-term impacts. This logic can be portrayed in many different ways, such as the model set out below. Figure 4 The intervention logic of a programme From this can be seen that the issues of intermediate outcomes and long-term impacts can only be addressed once the programme activities have been running for some time, and in the case of some targeted impacts, some time after the end of the activities. They are therefore not addressed in the interim evaluation. That being said, the interim evaluation will look at progress towards objectives a view to steering and fine-tuning the programme. It is also important to be clear on the nature of the longer-term objectives to ensure that necessary information (such as baselines and monitoring data) is being adequately collected. Of course, in reality the picture is somewhat more complex, multiple feedback loops and links between the elements multiple activities may be required for single outcomes, multiple outcomes may result from single activities. 6.2 Methodological concepts The key evaluation concepts are mapped on to the diagram above. These are: 31