Evaluation of the Food Quality and Safety priority within the 6 th Framework Programme

Transcription

1 Evaluation of the Food Quality and Safety priority within the 6 th Framework Programme D5 Final Evaluation Report Prepared for: European Commission Research Directorate-General Directorate E Biotechnologies, Agriculture, Food IDEA Consult in collaboration with COWI Denmark Brussels, October 5 th, 2010 IDEA Consult nv Phone: (+32) Kunstlaan 1-2, box 16 Fax: (+32) B 1210 Brussels

2 Table of Contents p. TABLE OF CONTENTS 2 1 EXECUTIVE SUMMARY Background and objectives of the study Evaluation tools Project implementation Partnerships Project outputs and results Longer term (expected) project results Contributions to the strategic objectives Recommendations 11 2 INTRODUCTION Introduction Context of the study Objectives of the study Content of this evaluation report Caveat 13 3 FOOD RESEARCH IN THE FRAMEWORK PROGRAMMES Introduction Food research under FP4 and FP Description food research under FP4 and FP Findings from FP4 and FP5 food research evaluations The 6 th Framework programme FP6 Thematic Priority 5: Food Quality and Safety Description Findings from FP6 food research evaluations Relevant findings from the overall ex post evaluation of FP Findings from the interim impact assessment of the NoEs under TP EVALUATION METHODOLOGY AND TOOLS Introduction Evaluation framework Intervention logic 25 2

3 4.3.1 Introduction FP6 TP5 Hierarchy of objectives Evaluation questions Evaluation tools Survey Case studies Workshop Caveat 31 5 PROJECT IMPLEMENTATION Introduction Sufficiency of funding Additional funding sources Management 39 6 PARTNERSHIPS Introduction Project collaboration Prior collaboration Continued collaboration Project size and composition Consortium size Partner mix Cooperation between old and new member states and third countries The extent and importance of international cooperation New Member States Third countries SME participation Gender balance 62 7 PROJECT OUTPUTS AND RESULTS Introduction Main objectives to participate Reach of the project objectives Reach of project objectives Suitability of the project Project outputs and results Main outputs and results Quality of the outputs Dissemination of project results Cost-effectiveness 85 3

4 8 LONGER TERM PROJECT RESULTS Introduction Influence on human mobility, training and employment Influence on policy Unexpected longer term results Implementation and exploitation of project results 91 9 CONTRIBUTIONS TO THE STRATEGIC OBJECTIVES Introduction Overall contributions towards strategic impact Findings Measures to enhance strategic impact Contributions towards establishing ERA Project outputs Longer-term impacts Achievement of strategic objectives Cross-country links Stability/renewal networks Contributions towards European competitiveness and scientific performance Longer term impacts on competitiveness and scientific performance Scientific and technological impact on knowledge producers and users Impacts through knowledge transfer and intellectual property Contributions towards meeting Community social objectives RECOMMENDATIONS Programme design Scope and focus Link with policy agenda Public good character of research outputs and industrial involvement Flexibility Project implementation Involvement of partner-groups Duration of the projects Programme and project management Follow-up activities Monitoring 122 4

5 1 EXECUTIVE SUMMARY 1.1 Background and objectives of the study Over the past decade, food quality and safety has grown into a significant area of public and media concern due to numerous food scares and a loss of consumer trust in the food chain. The Food Quality and Safety Thematic Priority 5 (TP5) of the Sixth Framework Programme (FP6) was established to specifically address this problem. The driving force behind TP5 was the need to improve the health and well-being of European citizens through higher quality food and improved control of food production and related environmental factors. The Biotechnology Agriculture and Food Directorate of the Directorate General for Research (DG RTD) wished to perform an ex-post impact assessment of its past research activities under FP6. IDEA Consult in cooperation with COWI Denmark was selected to carry out this study. The overall strategic objective of the study is: To evaluate the achievements and impacts of projects funded under Thematic Priority 5 (Food Quality and Safety) of the 6 th Framework Programme ( ), including the specific role of SMEs and International Cooperation. The more operational objectives of this evaluation can be described as follows: 1. To obtain evidence on the impact of TP5 under FP6 on their target groups and the impact of the project output on society; 2. To verify the relevance, effectiveness, efficiency, utility and added value of TP5 implemented under FP6; 3. To identify ways of improving and strengthening the programme s impacts; 4. To develop indicators for the monitoring of the 5 th priority impacts under the Framework Programmes for future assessment purposes. The table below provides some key statistics on TP5 within FP6. Table 1 1: Key statistics FP6 - Thematic Priority 5 Instruments Projects Participants Average number of participants per project Funding M Average funding per project (M ) Integrated Projects % % % 12.5 Networks of Excellence % % % 13.3 Specific Targeted Research Projects % % % 2.6 Coordination Actions 7 3.9% 214 7% % 1.2 Specific Support Actions % % % 0.5 Total % % % 4.1 Source: Idea Consult on the basis of the FP6 Dbase and updated information of project coordinators 5

6 1.2 Evaluation tools To assess the overall impact of TP5, we collected information from the participants in two rounds: by means of a survey as well as through case studies. We have set up an online survey addressed to all participants in TP5. We received 762 responses which implied a response rate compared to the net sample of 30.5%. In order to get a better insight into the (potential) impact of the projects, 30 case studies were carried out, in which 117 participants were interviewed. We further selected 5 experts to present their opinion on the scientific outputs and (potential) impacts of the projects. The results from the case studies have been analysed in two complementary ways: by a quantitative horizontal cross case analysis as well as by a qualitative vertical reading of all case study reports. In the final stage of the project we conducted a workshop with food research experts to gather their views on the preliminary findings of the study and to collect possible ideas and viewpoints regarding recommendations for strengthening the expected impact of similar future programmes. Note that we do not consider this study to be an ex-post impact assessment per se. We prefer to call it an evaluation study to indicate the broader scope of the study. We did study the (potential) impact of the projects but the possibilities to do so were limited, as the TP5 projects subject to this impact assessment have only ended recently or are still ongoing. 1.3 Project implementation According to about 3 out of 4 survey respondents and case study interviewees the EC-funding was in general sufficient both at the project level as for their own organisation. Partners that are most susceptible to budgetary insufficiencies are those partners involved in the coordination (especially in larger consortia), those involved for only a minor task and also SMEs. The private participants covered the costs that were not funded by the EC mainly by internal financial sources. In some of the projects, insufficient funding was solved by reallocating the budget from the partner with a budgetary shortage to a partner with a budgetary surplus. Further, in some organisations (mostly higher education institutes or research organisations) uncovered costs were funded indirectly via extra efforts from staff. Private investment from the companies was rather limited: although companies are responsible for nearly a quarter of the number of participations, they account for just 10 % if one considers their relative share in the EC contribution. This difference can only partially be explained by different cost rules. Companies also represent only 6% of the coordinators. Overall, 91% of all the survey-respondents agree that the project has been or is being managed and coordinated efficiently. The participants in the case studies confirm this positive assessment. The following issues, which are under the control of the consortium, have an enhancing influence on the efficiency of the management: a stable coordination team, previous 6

7 coordination experience, a high transparency and information-sharing, good communication and a substantial involvement of all partners in the consortium. In general, the survey-respondents as well as the case study interviewees do not support the idea that EC rules and practices enhance the efficiency of the project management; rather, they think these rules have a neutral impact or even the opposite. Most hampering are the financial rules and reporting requirements and the time required by the EC to implement contractually agreed payments. These results confirm the need for further simplification of the Framework Programmes, which has been taken up, at least partially, under FP Partnerships Overall, there is evidence of extensive collaboration both before and after the FP6 TP5 projects, also with the new partners. Prior collaboration has displayed a number of benefits to increase the success of the projects, and continued collaboration seems to include both existing and new partners of all types and countries (via FPs). The project size has been optimal in most cases. When not, project consortia were considered to be too large. This was primarily the case in the Integrated Projects. The statistics comparing the average number of contractors across thematic area indeed show that the consortia for the IPs under TP5 were by far the largest. The case study results show that there are negative consequences when projects are too large. With respect to the partner mix, a substantial part of the survey respondents indicated that large companies, and to a lesser extent SMEs, are underrepresented in the consortia. In cases where there are too many partners involved in the project consortium, this is mainly due to an overrepresentation of higher education institutes and research organizations. In terms of country representation we conclude that the majority of the survey respondents from the different country groups consider the country distribution of the consortia as optimal. The case studies have shed further light on the importance of securing a good country mix among the participants, for instance because there are food quality and safety issues that differ between geographical regions. The benefit of comprising such countryspecific differences is the ability to analyse how such differences affect food quality and safety. The international cooperation has been extensive (measured in cross border linkages), and the importance of this type of cooperation was very high. With about 10% of the participants, New Member States were represented well above average under TP5 compared to the other TPs, though their budget share was much smaller (5 %). While their participation was valuable, in some cases they lacked competencies and resources. The New Member States role was as followers rather than leaders. The importance of third country particiation was not valued as high as international cooperation in general, but still their participation was considered to be mostly optimal. The benefits to European projects from third country participants resources and knowledge was high, as well as benefits for the 7

8 third country partners. According to the respondents to the survey, the international standard of third country participants was high. Compared to the other thematic priorities, the Business Enterprise Sector and the SMEs in particular, were underrepresented as participants in TP5. Nevertheless, the involvement of SMEs is regarded as important to achieve the project objectives. SME participation was most successful in IPs. The available statistics show that TP5 attracts a high percentage of female researchers and in leading positions. It is therefore no surprise that gender issues played a minor role in the case study projects. 1.5 Project outputs and results The project objectives both at the level of the project and at the level of the organisation were (are expected to be) reached to a high extent, according to the survey respondents. Some of the projects among the case studies succeeded their project objectives. When projects (partially) failed to reach their objectives, the reasons mentioned were: (1) the rigidity of the FPprogramme, (2) cuts in the budget, (3) unexpected costs, (4) the transfer to industry and implementation of the results by industry taking time and (5) differences in interests. When we differentiate by type of organisation, the higher education institutes and the research organisations show a stronger tendency towards a complete reach of their organisation s project objectives opposite to the companies. The pooling of complementary knowledge and qualified personnel are the most important factors that contribute to the success of projects. Generally speaking, all project features identified as important are in accordance with the traditional European Added Value arguments justifying intervention in the field of research at EU level. Of the indicated project features, the pooling of IPRs and the sharing of risks contributed the least to the achievement of successful projects. In 90% of the cases the chosen type of instrument was considered to be the most suitable type. In most of the projects, also the quality of the research approach was considered to be high (40%) to very high (45%), with the interviewees from research organisations and higher education institutes being most positive. Scientific papers or publications in refereed journals and books are the most important output. Based on the data from our 30 case studies, we find that, on average, IPs produce the most peer-reviewed publications (129 per project), followed closely by NoEs (120). On average, 25 articles were published per STREP project. Other important outputs are: (1) new, improved or integrated tools, techniques, models, (2) information systems for data-sharing and (3) new or improved codes, standards and protocols. The more exploitable and commerciable outputs, such as demonstrators and prototypes, patents and new products or services are mentioned less by the survey respondents. Among the IP projects in the cases, on average, there had only been 2 patent applications while the NoEs that were analysed didn t generate patents. We therefore conclude that outputs were mainly research-focused and less industry-oriented. 8

9 The expert assessment of the cases, in terms of usefulness and quality of the actual research carried out during the project s lifetime, was high to very high. The relative citation rate of TP5 scores higher than one, meaning that TP5 lead scientists publications have a greater citation performance than all articles published in the same group of journals. However, TP5 ranks lower than most other Thematic Priorities. Dissemination happened/is happening to a medium or high extent. This dissemination is also effective as confirmed by 91% of the respondents in the survey and by most of the case study interviewees. In terms of geographic scope, the dissemination activities mainly reached EU15 countries, followed by EU10 countries. Third countries fell within the scope of the dissemination activities according to nearly half of the respondents, a percentage which is, surprisingly, higher than for the Candidate Countries (ticked by 43% of the respondents). The case studies results are much in line with the survey results. On average, the participants that responded to the survey considered the benefits higher than the costs: 59% of the respondents indicated the benefits were (much) higher than the costs, while only 9%% indicated the costs were (much) higher than the benefits. 1.6 Longer term (expected) project results Most of the 30 analysed cases have taken one or more actions related to human mobility and training, ranging from training courses and workshops to staff exchange and mobility grants. In the survey, both companies as well as the research oriented organisations reported, on average, a (tendency to a) large increase of their knowledge base. The influence of project participation on the quality of human resources was estimated lower, in particular by the company respondents. The longer term influence of project participation on employment within the companies is limited, according to the survey respondents. The interview results from the case studies show that in most of the projects the employment in the partner-organisations increased during the project. But a more permanent employment, however, is not always assured. Half of the case study projects provided some evidence of influence on pending or ongoing policy, with IPs seeming to have relatively more influence on policy compared to NoEs and STREPs. The direct influence on policy, however, is limited. The most common type of influence at the national level is the involvement in policy bodies & advisory boards and strategic action plans. In addition, some projects have an indirect influence on policy via EFSA 1 (policy dialogue, incorporation of results in guidelines,...) and the CIAA European Food Safety Authority Confederation of the food and drink industries of the European Union 9

10 As additional results, the survey-respondents mentioned: (1) the extension of their networks also outside the project team as such, (2) more structural cooperation/cluster initiatives that resulted from the network in the context of the project they participated in, (3) changes in the education process and curricula, (4) improved project management experience and (5) other economic results. According to the survey-respondents, scientific papers and publications in refereed journals are most likely to contribute to the follow-up and valorisation of the project results. The more applied outputs such as demonstrators, prototypes, patents and new products have a lower contribution to the follow-up and valorisation of results. There is uncertainty regarding the (potential) development by industrial partners and the (potential) licensing out of the project results. Only a very limited number of project results are already or are expected to be developed by industrial partners. In terms of potential economic impact, the experts also gave an overall medium level assessment of the cases. In many projects there is potential for an economic impact. However these require additional work that is dependent on additional resources. 1.7 Contributions to the strategic objectives The participants in the survey were asked about the contribution of their project towards the strategic objectives of Thematic Priority 5 Food quality and safety. These proposed potential impacts were inspired by the hierarchy of objectives and effects developed in the first stage of the evaluation. According to the survey respondents and case study interviewees, the projects contributed most to the realisation of the following objectives, which was also confirmed by the experts: 1. Exchange of knowledge and expertise in the area of food quality and safety with research partners across the world; 2. Improved collaboration and coordination in the area of food quality and safety; 3. The development of scientific and technical excellence in the area of food quality and safety; 4. The strengthening of the scientific and technological bases of the Community food quality and safety industry. When we rank the strategic objectives of TP5 in terms of the contribution that the projects delivered to these objectives, we obtain the following list (ranked by order of importance): 1. Scientific and technological impact on knowledge producers and users; 2. The impact on diffusion and exploitation in terms of knowledge transfer and intellectual property; 3. Socio-economic impact of research activities. The experts have been very positive towards the 23 projects they evaluated and all but three projects received a high or very high overall assessment of their contribution to the strategic and specific objectives of TP5. It is notable that the STREPs lie considerably below the average assessment. 10

11 1.8 Recommendations Programme design 1. More explicit programme design 2. More strategic programme management and less project management 3. Feasibility and compatibility check of additional programme objectives 4. Better balance between starting up new research themes and continuation of themes across FPs 5. Minimal linkages between different topics within IPs 6. More focus on integrational research 7. Engage with policy development 8. Increase potential for policy impact 9. The public good character of (some of) the research outputs should be respected Project implementation 10. The involvement of industry (and SMEs) should be considered as an opportunity and not forced as a requirement 11. Consider alternative ways for industrial inclusion 12. Provide more room for adjustments to the research agenda in the course of the project 13. Reconsidering the involvement of partners responsible for small tasks 14. Monitoring of NMS-involvement 15. Reconsideration of project-duration Programme and project management 16. Further simplification of administration and support for coordinators 17. Revision of the role of the co-ordinator Follow-up activities 18. Increase budgets for dissemination by top-up grants for successful projects 19. Invest in output translation 20. Inclusion of exit strategies Monitoring 21. Additional information collection, at the right time, to be able to evaluate and monitor the effects and impacts 11

12 2 INTRODUCTION 2.1 Introduction In this chapter we introduce the context and objectives of the study and present the content of this evaluation report. 2.2 Context of the study Over the past decade, food quality and safety has grown into a significant area of public and media concern due to numerous food scares and a loss of consumer trust in the food chain. The Food Quality and Safety Thematic Priority 5 (TP5) 3 of the Sixth Framework Programme (FP6) was established to specifically address this problem. The driving force behind TP5 was the need to improve the health and well-being of European citizens through higher quality food and improved control of food production and related environmental factors. The programme uniquely identified research needs to target major consumer issues along the production chain associated with primary production, animal feed, safety, nutrition, processing, distribution, consumption and environmental health risks and traceability, and which reflected a "fork to farm" approach as opposed to the traditional mechanism of farm to fork driven by intensive production. The Biotechnology Agriculture and Food Directorate of the Directorate General for Research (DG RTD) wished to perform an ex-post impact assessment of its past research activities under FP6. IDEA Consult in cooperation with COWI Denmark was selected to carry out this study. 2.3 Objectives of the study The overall strategic objective of the study is: To evaluate the achievements and impacts of projects funded under Thematic Priority 5 (Food Quality and Safety) of the 6 th Framework Programme ( ), including the specific role of SMEs and International Cooperation. The more operational objectives of this evaluation can be described as follows: 5. To obtain evidence on the impact of the 5 th Priority under FP6 on their target groups and the impact of the project output on society; 6. To verify the relevance, effectiveness, efficiency, utility and added value of the 5 th Priority implemented under FP6; 7. To identify ways of improving and strengthening the programme s impacts; 8. To develop indicators for the monitoring of the 5 th priority impacts under the Framework Programmes for future assessment purposes

13 2.4 Content of this evaluation report The third chapter provides background information on food research in the Framework Programmes, particularly FP6. In Chapter 4, we present the evaluation methodology and the tools we applied. In the next chapters we present our answers to the evaluation questions. Chapter 5 deals with the project implementation. Chapter 6 is about the partnerships. In Chapter 7 we discuss the project outputs and results. Chapter 8 continues with a discussion on the longer term project results while Chapter 9 considers the contribution of the projects to the strategic objectives of the Food priority within FP6. In Chapter 10 we present our recommendations. Note that this evalution report is accompanied by two other reports. The background report contains much more information on the background of food research in the Framework Programmes, presents in full detail the methodologies that have been applied and provides all survey results as well as a detailed analysis of the case study results. Therefore we sometimes refer in this evaluation report to the background report for more detail. The thirty case studies are presented in a separate Annex to this evaluation report. 2.5 Caveat The main objective of this project was To evaluate the achievements and impacts of projects funded under Thematic Priority 5 (Food Quality and Safety) of the 6th Framework Programme ( ), including the specific role of SMEs and International Cooperation. However, we do not consider this study to be an ex-post impact assessment per se. We prefer to call it an evaluation study to indicate the broader scope of the study. We did study the (potential) impact of the projects funded under Thematic Priority 5 (Food Quality and Safety) but the possibilities to do so were limited in particular because of the timing. As the TP5 projects subject to this impact assessment have only ended recently or are still ongoing, the bulk of the impacts have not materialised yet and could therefore not be analysed. As a consequence, when we present results on impact, we are dealing more precisely, most of the time, with potential impacts. Assessing the real impact can only be done within a number of years and by questioning the beneficiaries. Suggestions to do so are included under the recommendations. 13

14 3 FOOD RESEARCH IN THE FRAMEWORK PROGRAMMES 3.1 Introduction This chapter provides background on: food research under FP4 and FP5, as well as findings of relevant ex-post evaluations, the 6 th Framework Programme in general and on findings of the overall FP6 ex-post evaluation, food research under FP6, as well as on findings of the recently completed study on FP6 food-related NoEs. Much more background information in included in the background report. 3.2 Food research under FP4 and FP Description food research under FP4 and FP5 The first FP in which food research was included in a major way was FP4 ( ), which had a budget of ECU 13,215 million. The most important FP4 programme including food research was the Agriculture and Fisheries (FAIR) programme with a budget of 739,5 Mecu (5.6 %). The main objective of the FAIR programme was the promotion and harmonisation of research in the major European primary production food and non-food sectors of agriculture, horticulture, forestry, fisheries and aquaculture. The programme sought to promote links between research and the input and processing industries, together with the rural activities, the end user and the consumer. The programme covered all aspects of the production and utilisation of biological raw materials, with the aim of developing new markets, products and processes for raw materials coming from agriculture, forestry and fisheries while supporting evolving Community policies in the fields of agriculture, fisheries, forestry, industry, energy and the environment. The FAIR programme covered the following thematic areas: (1) Area 1: Integrated Production and Processing Chains (15% of the budget); (2) Area 2: Scaling-up and Processing Methodologies (7%); (3) Area 3: Generic Science and Advanced Technologies for Nutritious Foods (16%); (4) Area 4: Agriculture, Forestry and Rural Development (37%); (5) Area 5: Fisheries and Aquaculture (17%); and (6) Area 6: Ethical, legal and social aspects ELSA (8%). In addition, there were horizontal activities such as demonstration, ethical, legal and social aspects, specific measures in support of SMEs and accompanying measures such as research training grants, support for conferences and dissemination of results were also covered by the programme. 14

15 The Fifth European Community Framework Programme covering Research, Technological Development and Demonstration activities (FP ) had a multi-theme structure, consisting of seven Specific Programmes, and a total budget of 13,700 Mio Euros. Four Specific Programmes were Thematic Programmes: (1) Quality of Life and management of living resources (Quality of Life); (2) User-friendly information society (IST); (3) Competitive and sustainable growth (GROWTH); and (4) Energy, environment and sustainable development (EESD). The three other Specific Programmes were Horizontal Programmes, which underpinned and complemented the Thematic Programmes by responding to common needs across all research areas. The Quality of Life programme was built around 6 specific key actions: (1) Food, Nutrition and Health; (2) Control of Infectious Diseases; (3) The "Cell Factory"; (4) Environment and Health; (5) Sustainable Agriculture, Fisheries and Forestry, and Integrated Development of Rural Areas including Mountain Areas; (6) The Ageing Population and Disabilities. These were goal-oriented and problem solving. They were targeted at identifiable socio-economic and market needs. A unique feature of key actions was their response to Community policy objectives. The objective of the key action Food, Nutrition and Health was to promote the development of knowledge, technologies and methods, including pre-normative aspects, based on multidisciplinary approaches to produce a safe, healthy, balanced and varied food supply for consumers covering the whole food chain. Another key action relevant from the perspective of FP6 TP5 was key action 5 (Sustainable agriculture, fisheries and forestry and integrated development of rural areas including mountain areas). The aim was to develop the knowledge and technologies needed for the production and exploitation of living resources, including forests, covering the whole production chain, adapted to recent adjustments in the common agricultural and fisheries policies, while also providing the scientific basis for Community regulations and standards. Similarly, the aim was to promote the multipurpose role of forests and the sustainable management and utilisation of forest resources as an integral factor of rural development Findings from FP4 and FP5 food research evaluations As shown in the previous section, food quality and safety research was already supported under FP4 and FP5. The purpose here is to highlight some findings from related ex-post evaluations that can set the scene for the present evaluation. These concern reasons for success or failure in achieving project objectives as well as specific achievements in terms of SME and international cooperation. Since only some of the instruments implemented under FP6 were also implemented under FP4 and FP5, the direct applicability of results is somewhat limited. Despite differences in approach and focus, several evaluations appear to have two observations in common. The first observation concerns the lack of dissemination of research results, especially to industry and to the wider public. One study pointed towards a communication gap. Another study identified a lack of communication skills as regards the dissemination of knowledge, especially to the media and the public. These findings are also in line with those of the recently completed FP6 TP5 NoE study. 15

16 Another frequent observation is that technology transfer through researcher mobility ends almost immediately once a project is completed. Two evaluations specifically recommended that more attention be paid to increasing the extent of training and to working towards retaining researchers in their research careers, as well as to the period right after the project ends and this to ensure that researchers do not directly move on to other work-areas. As far as industrial participation in the FP is concerned, communication and collaboration with industry appears to be a continuing challenge, an important issue being the dissemination and enhancement of industrial exploitation of European research. Demonstration projects involving both industry and academia have potential, but in general more consideration should be given to technology transfer and intellectual property rights. Industry has demonstrated a growing interest in research activities. One recommendation from the evaluations is that the industrial orientation of and participation in the FP must be enhanced through an increased focus on industrial relevance and leadership in programmes aiming at innovation and competitiveness. Several studies made interesting observations with regard to SMEs. Collaboration between researchers and industry - in particular SMEs was found to add value to projects. Under FP4, some particular support tools for SMEs were implemented: the Exploratory Awards and the Co-operative Research for Technology (CRAFT). These tools supported the participation of SMEs. The same report mentioned the Innovation Relay Centres as effective instruments for the transfer and exploitation of new technologies. One evaluation mentioned funding delays as obstacles to SME participation as SMEs do not have the necessary financial means to continue operating for a long period of time when payments are delayed. In the same vein, another evaluation pointed towards the need for increasing the attractiveness of participation, especially for high-tech SMEs. One reason for low SME participation in the new instruments was found to be the relative unfamiliarity of these. International cooperation was not widely discussed in previous evaluations, except for the impact assessment of the Specific Programme International RTD Cooperation under FP5 ( ). The reason was that under FP4 and FP5, separate International Cooperation programmes existed in addition to the thematic research programmes. The studies not specifically addressing the issue of international cooperation generally observed that the level of international cooperation has been increasing. With regard to New Member State participation in the FP, they identified some challenges. These are of a more general nature - relating to lack of capacity/capability, lack of networks, lack of capital and equipment etc. and not directly related to the structure of the FPs. The study that specifically focused on international cooperation made several relevant observations. It underlined in particular the need to focus on the practical implementation of international cooperation. The study mentioned, for instance, that a number of Science & Technology agreements signed by the EU with third countries were not being implemented in practice. It also mentioned that thematic areas should be considered more in detail to identify the most relevant INCO approach - e.g. mutual interests, needs-driven cooperation, etc. With regard to impact, the evaluations generally conclude that EU funding programmes add value to European research, expand the European knowledge base, and have strengthened the European RTD landscape. They have been instrumental in strengthening European capabilities and have contributed to the 16

17 generation and diffusion of new knowledge. EU funding also contributes to largescale facilities serving the research community. Networks have contributed greatly to transnational collaboration and training activities have increased the mobility of the researchers across borders. EU funding furthermore contributed to the inclusion of regions involved with research to a lesser extent, and the inclusion of young and female researchers. Another view is that the programmes are perhaps somewhat too ambitious compared to the level of funding, and that perhaps there is too much control of the overall funding strategy, leaving little room for researchers to take new initiatives regarding future research directions. 3.3 The 6 th Framework programme The Framework Programmes have had to serve two main strategic objectives: strengthening the scientific and technological bases of industry and encouraging its international competitiveness while promoting research activities in support of other EU policies. The main focus of the 6 th Framework Programme (FP6) was the creation of a European Research Area (ERA) as a vision for the future of research in Europe. It aimed at scientific excellence, improved competitiveness and innovation through the promotion of increased co-operation, greater complementarity and improved co-ordination between relevant actors, at all levels. FP6 was an important building block in the EU's drive to increase its overall research effort (1.9% of GDP in 2002) to as close to 3% of GDP as possible by In this respect, FP6 should have a structuring effect on research and technological development in Europe, including the Member States, Associated countries and other countries. The overall budget (EC) covering the period was 17.5 billion making up 3.9% of the Union s total budget (2001), and 6% of the EU's public (civilian) research budget. FP6 was composed of three main blocks of activities grouped in two specific programmes ( Integrating and Strengthening the ERA and Structuring the ERA ), plus a third specific programme on nuclear research. The first block Focusing and integrating European research accounted for the bulk of the FP6 budget. This block included research in 7 key priority areas which were allocated 69% of the total budget excluding Euratom. They covered those areas of scientific or technological competence where the EU in the medium term intended to become the most competitive and dynamic, knowledge-based economy in the world capable of sustainable economic growth with more and better jobs and greater social cohesion. As one of the measures to implement the international dimension of FP6, this block was open to participation by organisations from third countries with substantial funding included in the budget. These seven priorities were: Life sciences, Genomics and Biotechnology for Health; Information Society Technologies; Nano-technologies and nano-sciences, knowledge-based multifunctional materials, new production processes and devices; Aeronautics and Space; Food Quality and Safety; 17

18 Sustainable Development, Global Change and Ecosystems; Citizens and Governance in a knowledge-based society. Thematic research was implemented through five different instruments: Integrated projects (IPs) Specific Targeted Research Projects (STREPs) Networks of Excellence (NoE) Coordination Actions (CAs), Specific Support Actions (SSAs) Each instrument had its own rationale and key characteristics which are described in full detail in the background report. The following box provides a summary. The background report also presents a brief overview of some of the findings of the overall FP6 ex-post evaluation. They set the scene and provide a useful benchmark for the FP6 Thematic Priority 5 impact assessment. Box 1: Instruments under FP6 Network of Excellence (NoE) Implemented for the first time under FP6 (and continued under FP7), NoEs are multipartner projects aimed at strengthening scientific and technological excellence on a particular research topic by integrating at European level the critical mass of resources and expertise needed to provide European leadership and to be a world force in a given domain. This expertise had to be networked around a joint programme of activities aimed primarily at creating a progressive and durable integration of research capacities of network partners while at the same time advancing knowledge on the topic. Participating institutions have to invest seriously in structural change aiming at a durable integration of their research capacities. This requires the commitment of all levels of decision-making in an institution, including top management, supervising and financing bodies. Therefore, the main result should be a durable restructuring and reshaping of the way research is carried out in Europe in a given area. Integrated Project (IP) Similarly to NoEs, IPs were also implemented for the first time under FP6 (and are continued under FP7). IPs are multipartner projects to support objective-driven research, where the primary deliverable was to generate the knowledge required to implement the thematic priorities. IPs should bring together a critical mass of resources to reach ambitious goals aimed either at increasing Europe s competitiveness or at addressing major societal needs. They must contain a research component and may contain technological development and demonstration components, as appropriate, as well as perhaps a training component. Integration within an integrated project could take several forms: vertical integration of the full value-chain of stakeholders; horizontal integration of a range of multidisciplinary activities; activity integration: integrating various research activities from fundamental to applied research and with other types of activity; sectoral integration of actors from private and public sector research organisations and in particular between academia and industry, including SMEs; and financial integration of public and private funding. The effective management of knowledge and its dissemination and transfer, was an essential feature of each integrated project together with the analysis and assessment of the technologies developed and of the factors relating to their exploitation, where relevant. STREPs 18

19 Specific Targeted Research Projects (STREPs) are multipartner research, demonstration or innovation projects. They were considered to be an evolved form of the shared-cost RTD projects and demonstration projects used in FP5. Their purpose was to support research, technological development and demonstration or innovation activities of a more limited scope and ambition than IPs. Coordination Action (CA) CAs were multi-partner actions intended to promote and support the networking and coordination of research and innovation activities. They were initiated under FP6 as a reinforced form of the concerted actions/thematic networks used in FP5. They covered the definition, organisation and management of joint or common initiatives as well as activities such as the organisation of conferences, meetings, the carrying out of studies, exchanges of personnel, the exchange and dissemination of good practice, setting up common information systems and expert groups. Specific Support Action (SSA) Being the FP6 version of the accompanying measures under FP5, support activities were more limited in scope. These projects aimed to contribute actively to the implementation of activities of the work programme, the analysis and dissemination of results or the preparation of future activities, with a view to enabling the Community to achieve or define its RTD strategic objectives. 3.4 FP6 Thematic Priority 5: Food Quality and Safety Description As indicated before in the introduction, the driving force for setting up TP5 in the Sixth Research Framework Programme (FP6) was the need to improve the health and well-being of European citizens through higher quality food and improved control of food production and related environmental factors. The programme uniquely identified research needs to target major consumer issues along the production chain associated with primary production, animal feed, safety, nutrition, processing, distribution, consumption and environmental health risks and traceability, and which reflected a top-down or "fork to farm" approach as opposed to the traditional mechanism of farm to fork driven by intensive production. In addition to combining production, processing, nutritional and analytical expertise, projects drew on expertise from areas such as genomics, medicine, information technologies, production engineering, ethics, environmental, economic and the social sciences, to achieve their aims. In doing so, the strategy provided a unique approach in the development of a total food chain approach to food quality and safety. Accordingly, over this period the food sector itself has shifted its focus more towards consumer needs, resulting in improved links between production, distribution and consumption. This has helped somewhat in allaying concerns and restoring consumer confidence across Europe. The research funded in TP5 however was supposed to help consolidate this focus and reestablish and maintain the competitive edge of the European Food industry by introducing new leading edge technologies and sustainable practices into the growing and processing of food which will be of benefit to the health and well being of the consumer, and the environment. The 5 th priority represented 6.1% of the total budget allocated to the seven thematic priority areas and 4.2% of the total FP6 budget (excluding Euratom). 19

20 In terms of themes, the programme encompassed many of the research topics funded by the previous FP5 "Quality of Life programme" which ran from 1998 to However in the interest of focussing efforts and resources upon the particular crucial issue of food safety and health, some aspects of the previous programmes portfolio were dropped (e.g. non-food research, forestry and animal health) or dealt with elsewhere (e.g. under the eight priority which dealt with support to fisheries and agricultural policy). The TP5 programme was thus shaped around the whole food chain concept from farmer to consumer and was divided up over eight scientific areas in response to the overall programme objectives: FOOD 1: Total food chain; FOOD 2: Epidemiology of food-related diseases and allergies; FOOD 3: Impact of food on health; FOOD 4: Traceability processes all along the production chain; FOOD 5: Methods of analysis, detection and control; FOOD 6: Safer and environmentally friendly production methods and technologies and healthier foodstuffs; FOOD 7: Impact of animal feed on human health; FOOD 8: Environmental health risks. While in terms of themes Priority 5 encompassed many of the research topics funded by FP5, there was an important shift in terms of instruments with a move towards funding larger, more ambitious projects using the new funding instruments designed for FP6, as well as retaining some support for smaller projects through more traditional instruments (see infra). FP6 Thematic Priority 5 was implemented through four main calls for proposals (in 2002, 2003, 2004 and 2005) and ten individual deadlines. In these calls, each specific topic was linked to a particular instrument, or choice of instruments, and it was not possible to secure funding with traditional instruments for small projects in areas that were already covered by topics inviting integrated project or network of excellence proposals. International co-operation was an important aspect and strategic objective of this programme: participation was open to nearly all third countries world-wide and dedicated support was foreseen for INCO-target countries. Participants from INCO target countries could be funded within the limits of available budget earmarked for international co-operation (indicative 15.9 MEUR for Food Quality and Safety) by means of specific support actions. Small and medium-sized enterprises (SMEs) were a major focus of the Sixth Framework Programme. Project consortia could include SMEs as technology providers and research performers, service providers or technology users. They could also be involved in exploitation, dissemination or training activities. SMEs were expected to play a particularly important role in integrated projects and networks of excellence. It was possible to reserve a part of the budget to include new SME partners, as appropriate, as the project evolved. The Commission moreover financed a number of support actions with the objective of increasing SME participation. 20

21 3.4.2 Findings from FP6 food research evaluations Relevant findings from the overall ex post evaluation of FP6 The expert group that made an overall evaluation of FP6 mentions in their report 4 that projects in the Food quality and safety research area produced advanced knowledge about how to improve quality and safety of food. In many cases, the networked institutions involved, established common databases and methods so that the multinational system for ensuring safe food is coherent often outside as well as inside the EU. A result of the strong focus on safety, however, was limited industrial participation. While industry was clearly engaged in, and will be affected by, the safety work, firms tend to be more interested in exploring knowledge that will in future enable them to introduce new and improved product and process technologies. FP6 did not provide much opportunity for this type of exploratory research, focusing on finding bases for new improved processes, products and solutions Findings from the interim impact assessment of the NoEs under TP5 In addition to this overall evaluation, an interim impact assessment 5 of the Networks of Excellence under the Food Quality and Safety Programme was completed by COWI in March A distinction has been made between general lessons on the one hand and specific lessons regarding SME participation and international cooperation on the other hand. General findings The impact assessment concluded that all eight food related NoEs studied had been successful in integrating research institutions. The NoE instrument has for the research topics addressed enabled a shift from a situation in which European research institutions collaborated to a limited degree only to a situation marked by much closer research relationships. The NoE instrument has supported this integration. It has made it possible for research institutions that did not know each other very well or were uncertain about potential research synergies to actually meet and investigate them. It also made it possible to organise many personal networking activities and joint research activities, which contributed significantly to confidence building between researchers from different NoE partners with respect to the sharing of knowledge and findings. The bringing together of different research institutions has also resulted in the compilation of datasets, partly through agreement on standards for data provision and partly through the compilation of data in databases or virtual laboratories accessible to NoE partners. Such larger datasets have enabled the achievement of more statistically sound research results. Several of the integration achievements are likely to be durable. Yet none of the eight NoE structural set-ups will continue unchanged after the end of the EC support period. Most or even all NoEs will continue as some kind of legal entity, primarily as associations. These associations will be kind of 'light' versions of the present NoEs operating with a slimmer management setup, possibly with fewer partners, and with fewer activities, the main continuing ones possibly being the 4 5 Evaluation of the Sixth Framework Programmes for Research and Technological Development, Report of the expert group, (February 2009). COWI (2009), Food Quality and Safety programme: Interim Impact Assessment Networks of Excellence, prepared for EC DG Research. 21

22 maintenance of a common website and the organisation of an annual (NoE) conference. The NoEs have produced two types of research results: integrational research results and traditional research results. The integrational research results comprise the development of new (joint) research methods/analytical approaches, (joint) standards/norms, databanks/biobanks, and guidelines for further (joint) research. Traditional research results comprise scientific publications and papers as well as patents and other property rights. An integrational research result stems from collaborative work and forms the backbone for further joint research activities. As mentioned, it is based on trust and confidence acquired through giving access to each others' data sets, results, etc. It is, however, difficult to assess the excellence of the such research results. There is not much experience in carrying out integrational research and consequently not many benchmarks to compare with. The absence of such benchmarks also indicates that such scientific interaction would have been substantially more limited in the absence of NoEs. Findings regarding SME participation The participation of enterprises in NoEs has been limited. This can partially be explained with reference to the fact that most of the research undertaken was of a basic research nature and could not be commercialised immediately. Another reason was that it was difficult to integrate some enterprises within a NoE while excluding their competitors. The dissemination of research results to enterprises outside NoEs has been somewhat more successful. Yet many food enterprises - in particular SMEs - are not geared to making direct use of research results (the prospects for this are more promising in the health/pharmaceutical sector). SMEs and other enterprises have participated in NoEs in a more indirect manner. Industry representatives were invited to many of the main NoE events such as annual NoE conferences, scientific conferences, workshops and other events. In some cases, industry participants made presentations to scientific audiences. Other NoE events/meetings targeted industry directly. The degree of industry involvement differed substantially across NoEs depending on the nature and relevance to industry of the scientific field concerned. The external parts of NoE websites were also a key information source for enterprises and industry associations. The types of material typically available included newsletters and scientific publications. Many industry organisations, and sometimes also individual business enterprises, were on ing lists or received hard copy material by post. Findings regarding international cooperation The NoEs have contributed to an internationally acknowledged European Research Area (ERA) and the outside world, e.g. the United States, envies such accomplishments. The NoEs have furthermore established themselves as European research brands, i.e. NoE research results have been recognised internationally as making distinctive European contributions to science in their respective research fields. This is also reflected in the maintenance of NoE names when establishing new legal entities. However, NoEs have not been encouraged directly by the European Commission to integrate research institutions from, for instance, third countries. In this respect, NoE coordinators generally emphasise that it has been cumbersome 22

23 enough already to integrate EU research institutions, in particular those from the new Member States. Hence, there has not been much energy to pursue even further international outreach. 23

24 4 EVALUATION METHODOLOGY AND TOOLS 4.1 Introduction In this chapter we present the pillars of our evaluation methodology and the tools that have been applied in the course of the study. We start with an introduction of the evaluation framework that inspired our overall approach. Subsequently we present the hierarchy of objectives and effects that were constructed on the basis of desk research, exploratory interviews and a workshop with Commission officials. In the next section we list the evaluation questions, inspired by the hierarchy of objectives and the hierarchy of effects and structured by the evaluation topics. The main tools that we applied to provide an answer to these evaluation questions are detailed in the final section. We refer to the background report for a more extensive description of the methodology. 4.2 Evaluation framework In Figure 4-1 we present the framework that was used for this evaluation. It refers to the typical intervention logic when policy makers want to address a specific problem, like in this case the support of R&D in the field of Food Safety and Quality. When linking the needs of the stakeholders in the whole food chain (from consumer to farmer) with the objectives of this priority under FP6, we address the topic of relevance. In order to evaluate the effectiveness of the EC funding provided, we link the outputs as well as the results with the stated objectives of Priority 5 under FP6. Linking the projects and their outputs with the inputs will allow us to address the efficiency topic. The impact question will be defined by linking the results of the EC funding under this priority with the impact they induce. The utility links the impacts and outcomes with the needs of the stakeholders involved. Sustainability refers to the existence and especially the perpetuity of impacts that go beyond the direct effects. Finally the topic of added value refers to the added value of the thematic priority compared to other existing policy initiatives. 24

25 Figure 4-1: General framework of the evaluation Research Food Quality and Safety UTILITY IMPACT of FP6- Priority 5 RELEVANCE of FP6 Priority 5 of FP6 Priority 5 of FP6 Priority 5 Research Project EFFICIENCY EFFECTIVENESS In the context of this project, the focus was on the evaluation of the effectiveness and to a lesser extent, the impact and sustainability of the projects and programme (see infra). 4.3 Intervention logic Introduction To facilitate the link between what we evaluate and the objectives of Priority 5, a so-called hierarchy of objectives was defined (for a definition, see Box 2). Box 2: Definition of hierarchy of objectives The hierarchy of objectives is a tool that helps to analyze and communicate the objectives of a programme or other policy intervention. It organizes these objectives into different levels (strategic objectives, specific objectives and operational objectives) in the form of a hierarchy or a tree, thus showing the logical links between the various levels. Three main levels of objectives can be distinguished: 1. Strategic objectives (main strategic objectives at the global level as well as strategic (sub-)objectives at the intermediate level) represent the high-level objectives and are defined very generally. These have, for example, a direct link with the Lisbon Strategy. 2. The specific objectives indicate through which channels the strategic objectives can be reached. They have a (more) direct link with the core activities that take place under Priority 5. 25

26 3. Operational (activity-related) objectives are the lowest-level, most specific objectives and have a close link with the activities undertaken by the partners in the projects funded. In parallel to a hierarchy of objectives, a hierarchy of effects was established. As in the case of objectives, different levels of expected effects can be distinguished depending on how closely the effects can be linked to an action undertaken by the partners in projects funded in the context of Priority 5. As the following figure shows, the hierarchy of effects is closely linked to the hierarchy of objectives and follows a similar structure. Figure 4-2: Relationship between the hierarchy of objectives and the hierarchy of effects Source: IDEA Consult FP6 TP5 Hierarchy of objectives Based on our own understanding of FP6 Thematic Priority 5, which was achieved through desk research (FP6 decisions, relevant Work Programmes etc.) and exploratory interviews and based also on a workshop held on 17 June 2009 with EC officials thoroughly familiar with FP6 TP5, the intervention logic and hierarchy of objectives pertaining to Thematic Priority 5 was reconstructed at the level of strategic (global and intermediate) and specific objectives. The operational objectives were assumed to coincide with the characteristics of particular instruments. Global Strategic Objectives To improve the health and well-being of European citizens To promote sustainability and the preservation of the environment To increase the competitiveness of industry, promote economic growth and employment 26

27 Intermediate Strategic Objectives To contribute to a higher quality of food for European citizens To contribute to improved control of food production To contribute to improved control of related environmental factors To contribute to the development of an environmentally friendly production and distribution chain of safer, healthier and more varied food To contribute to an improved understanding of the link between food and health To contribute to improved control of food-related risks To contribute to improved control of health risks associated with environmental changes To contribute to innovation To strengthen the scientific and technological bases of the Community food quality and safety industry To strengthen Community food quality and safety policy To contribute to the development of scientific and technical excellence in the area of food quality and safety To contribute to the establishment of the ERA in the area of food quality and safety Specific Objectives To promote innovative, consumer-driven intra-european cross-border collaborative R&D in the area of food quality and safety characterised by European added value, critical mass, multi-disciplinarity, the involvement of a broad range of fork to farm stakeholders To promote researcher education, training and mobility in the area of food quality and safety To strengthen the technological capacities and international competitiveness of small enterprises in the area of food quality and safety and facilitate access to the best research and technology To involve third countries in Community research policy and ERA in the area of food quality and safety in the interest of obtaining mutual benefits To support specific EU policies (e.g. labelling, traceability, etc.) Once this particular hierarchy of objectives had been established it was comparatively easy to define the hierarchy of effects, which is the mirror image of the hierarchy of objectives (see background report). 27

28 4.3.3 Evaluation questions Table 4 1 categorises the evaluation questions listed in the Terms of Reference - as well as additional ones developed during the inception stage - by programme/project level and by evaluation topic. Table 4 1: Evaluation questions to check met final eversie van evaluatievragen Evaluation questions Programme level evaluation questions Relevance (1) Are the programme's research objectives still relevant? What would be the main recommendations for future programmes emanating from this impact analysis? Effectiveness (2) Have overall programme objectives been met? (3) What were the main projects outputs and results that were generated by the projects, and what is the overall quality and relevance (or usefulness ) of these outputs? (4) Did the activities contribute to developing a European Research Area? Are there any clear indications on emerging wider European structures, policies and programmes visible in the Member States as a result of these projects? (5) To which extent have the outputs and results been disseminated (also in terms of geographic scope) and how effective have the project dissemination actions been? (6) What role has private investment played in the projects especially that from small and medium enterprises? (7) What role has gender played in the overall projects with respect to participation and responsibility? Efficiency (8) Was the overall management and financial infrastructure supplied by the Commission of sufficient quality to deal with the correct enactment of the programme? (9) Was the project size and participation optimal especially with respect to the new instruments? (10) Was the level of funding adequate to reach the project objectives? (11) Was the level of funding adequate given the project tasks assigned to the organisation? (12) How did the participants cover the costs in the project that were not covered by the EC financial contribution? Utility (13) What has been the true motivation for the presence of private entities in the project? Sustainability (14) Has project participation/did project participation have an influence on employment during the project and have those new employees retained employment as a results of the project? (15) To which extent did the project partners collaborate before the project? (16) To which extent did the project partners continue their collaboration after the end of the project? 28

29 Evaluation questions Impact (17) Has there been any impact on European competitiveness and scientific performance through increased knowledge, capacities and infrastructures? (18) What have been the effects of the programme on Community social objectives such as quality of life, health, employment, the environment? Added value (19) Can any major gaps, failures or inconsistencies in the Framework Programme and/or in the Specific Programme be identified with respect to this thematic area? Project level evaluation questions Relevance (20) How relevant has each project been to pending or ongoing policy legislation, and if so in what way has the project been of influence? Effectiveness (21) Did the projects constitute the best way of obtaining the objectives set (in terms of chosen instrument, research approach,...)? (22) What have been the main objectives to participate? (23) Did the projects attain their original aims? (24) Did the projects produce unexpected results? (25) Were the individual project deliverables delivered on time? (26) Were the individual project deliverables easily accessible to peer groups? (27) To which extent has international cooperation with third countries been implemented in the projects and via which instruments? (28) Has international co-operation through the inclusion of third countries in the projects been of benefit to either the EU and/or the third party? (29) Were the third country participants of high international standard? (30) What has been the role of new member states in the projects? (31) What was the role of SMEs participating and what characterises their participation? (32) To which extent did SMEs participate in the projects and via which instruments? Efficiency (33) Were the projects carried out and managed efficiently? (34) Were the projects cost-effective? Impact (35) Have the projects obtained any results worthy of exploitation and dissemination? (36) What has been the effect of the projects on human mobility and training? (37) Which of the project ouputs contributed or most likely will contribute to the follow-up and valorisation of the project? 4.4 Evaluation tools To assess the overall impact of Priority 5 Food Quality and Safety of FP6, we collected information from the participants in two rounds: by means of a survey as well as through case studies. The background report describes in full detail the set up and execution of both the survey and the case studies. We further had a 29

30 workshop with food experts to discuss the draft results and conclusions from the study and to reflect on potential recommendations Survey We have set up an online survey addressed to all participants in thematic priority 5. The structure of the questionnaire included 5 sections: - Section 1: Characteristics of your organisation - Section 2: Funding, management and partnership - Section 3: Project outputs and impacts - Section 4: Contribution of the project to the strategic objectives of Thematic Priority 5 Food Quality and Safety - Section 5: Additional comments The response to the survey was satisfactory. We received 762 responses which implied a response rate compared to the net sample of 30.5% 6. The response analysis showed that there are in general only small differences between the original dbase (the TP5 FP6 population) and the group of survey respondents if we look at the distribution by country, instrument and thematic area. For these characteristics we detected no response bias. The relative share of the coordinators in the net response group was significantly higher than its relative share in the gross sample and the population. This bias can be explained by our targeted efforts to get answers from the coordinators. Furthermore, public research organisations were over-represented in the survey, while companies and private research organisations were underrepresented. At least one participant of each TP5 FP6 project was invited to participate in the survey, and thus included in the gross sample. Thus, all 181 TP5 projects were included in the gross sample. For 14 of these 181 projects, we have not received any answer. Consequently, the net response at the level of the project (not taking into account the number of respondents) was 92.3%. In other words, for 167 projects out of the total number of 181 projects, at least one project partner responded to the survey Case studies In order to get a better insight into the (potential) impact of projects that were conducted under the FOOD Priority of FP6, 30 case studies were carried out at the project level (8 NOE s, 15 IPs and 7 STREPs) 7. What we have been seeking to obtain from the case studies, was case-based, empirical evidence on the effects and (potential) impacts of participation on participants in projects financed under the FOOD priority and the (potential) longer-term impacts of project outputs on the strategic objectives of TP5. They complemented and deepened the empirical evidence gathered through the online surveys and at the same time broaden a qualitative dimension of the analysis. 6 7 In the background report we included an overview of 7 previous surveys among project participants in the context of ex-post evaluation studies of the 6 th Framework Programme with response rates between 9,2 and 32 %. Please see the background report for a detailed description of the case study approach. 30

31 We established a group of 8 countries from which projects were selected: Belgium, Denmark, Finland, France, Italy, Poland, Spain and the UK. We considered those projects that had at least been ongoing for 3 years, i.e. that started before February 2007, in order to allow for effects to materialise. For the same reason, we also made sure that the projects finish at the latest in participants were interviewed, almost 4 per project (on average 2 face-toface, 2 by telephone). We further selected 5 experts to present their expert opinions on the scientific outputs and (potential) impacts of the projects. These expert reviews were important inputs to the combined assessment of the projects and parts of the reviews were integrated into the case study reports being prepared for each of the case projects. The results from the case studies have been analysed in two complementary ways: by a quantitative horizontal cross case analysis as well as by a qualitative vertical reading of all case reports Workshop In the final stage of the project, on June 16 th 2010, we had a workshop in Brussels. The workshop intended to gather the views of the participating experts on the preliminary findings of the study and the conclusions that can be drawn. Secondly, the workshop aimed to collect possible ideas and viewpoints of the participants regarding recommendations for strengthening the expected impact of similar future programmes. The results of the workshop discussions have been integrated in the analysis of the survey and case study results and were also inspiring for the recommendations of the study. 4.5 Caveat The main objective of this project was To evaluate the achievements and impacts of projects funded under Thematic Priority 5 (Food Quality and Safety) of the 6th Framework Programme ( ), including the specific role of SMEs and International Cooperation. There are two central challenges in general to assessing impact: boundary judgment, i.e. deciding what effects to select for consideration, and attribution, i.e. what effect is due to what. Proving the impact of R&D-support in particular is also a very difficult exercise. The very nature of Framework Programme research, based typically on precompetitive collaborative research projects means that there are likely to be many steps, and complementary inputs downstream of the research, before commercial or other types of return and impact can be realised. Further still, the academic literature shows very clearly that there are severe measurement challenges when seeking to assess the totality of research impacts. These relate to both the problems of attribution, connecting impacts with the underpinning research, as well as with the many 'spillovers' which flow from a research action. Data availability, especially tracking research over the longer term is also a severely limiting factor. We do not consider this study to be an ex-post impact assessment per se. We prefer to call it an evaluation study to indicate the broader scope of the study. The evaluation addressed the following: 31

32 the delivery of project outputs, i.e. the degree to which the projects operational or activity-related objectives have been achieved; the results, i.e. the effectiveness of the projects in reaching their specific or direct objectives; and finally the impacts, i.e. the achievement of the projects intermediate and global strategic objectives (see below). In addition to assessing the achievement of project and programme objectives, we looked at the underlying factors leading to objective achievement, including suitability of size and composition of project consortia, management performance, appropriateness of project funding arrangements and resources, the costeffectiveness of the projects in turning resources into outputs and results, etc. Reaching these lower level objective is a condition sine qua non for reaching the higher level specific and strategic objectives. So we did study the (potential) impact of the projects funded under Thematic Priority 5 (Food Quality and Safety) but the possibilities to do so were limited in particular because of the timing. While a few impacts occur in the short, most impact only materialise in the medium to longer term, i.e. maybe three, five or even ten years after a project has completed. Economic impacts for instance will in particular in the food sector take a long time because the sector is mainly composed of manufacturing SMEs that pick up new innovations slower compared to larger companies. As the TP5 projects subject to this impact assessment have only ended recently or are still ongoing, the bulk of the impacts have not materialised yet and could therefore not be analysed. As a consequence, when we present results on impact, we are dealing more precisely, most of the time, with potential impacts. Moreover, ideally, strategic impact should be identified as perceived or experienced by the actual, potential beneficiaries and users. In this impact analysis we have for methodological and practical reasons opted to obtain the views of the project participants on realised and potential benefits as it was outside the scope of the assesment to involve potential future beneficiaries more broadly 8. The risk of this approach is that the impact measurements or scores provided by the respondents overestimate the potential future impacts. To sum up, the results on impact that we present are to be considered as an audit or assessment of research programme participants perceptions (mainly the researchers themselves) of the effectiveness of the research projects and their potential impact. Assessing the real impact can only be done within a number of years and by questioning the beneficiaries. Suggestions to do so are included under the recommendations. 8 Please note that we did interrogate the industrial partners in the projects, both in the context of the survey as well as during the interviews for the case studies. They can be considered as (first) potential commercial beneficiaries. 32

33 5 PROJECT IMPLEMENTATION 5.1 Introduction In this chapter, we will give an answer on the following evaluation questions: EQ 1: Was the level of funding adequate to reach the project objectives EQ 2: Was the level of funding adequate given the project tasks assigned to the organisation? EQ 3: How did the participants cover the costs in the project that were not covered by the EC financial contribution? EQ 4: What role has private investment played in the projects; especially that from small and medium enterprises? EQ 5: Were the correct procedures deployed to address the topic? EQ 6: Was the overall management and financial infrastructure supplied by the Commission of sufficient quality to deal with the correct enactment of the programme? The answers on these evaluation questions are discussed under the following sections: Sufficiency of funding Additional funding sources Project management and EC practices 5.2 Sufficiency of funding The results presented in Figure 5-1 give an answer to the questions (1) whether or not the survey-respondents considered the EC financial contribution to be sufficient given the overall project objectives and (2) whether or not they considered the specific part of the contribution granted to their organisation to be sufficient given the project tasks assigned to their organisation. 33

34 Figure 5-1: Sufficiency of EC-funding at the project (a) and organisation level (b) (n=751) 100% 90% 80% 27.3% 30.0% 70% 60% 50% 40% 30% 20% 10% 72.7% 70.0% No Yes 0% Sufficiency at the project level Sufficiency at the organisation level (a) Was the EC financial contribution to the project sufficient given the overall project objectives? (b) Was the share of the EC financial contribution granted to your organisation sufficient given the tasks assigned to your organisation under the project? Source: IDEA Consult on the basis of survey results Almost 73% of the survey respondents considered the EC financial contribution to the project to be sufficient. These results are confirmed by the case studies, with 70% of the interviewees considering the EC financial contribution to the project to be sufficient. The survey-results show only small differences between the different roles and the different types of organisations. By instrument, the survey-respondents participating in CAs seem to be the least convinced (with 64.7% reported yes ). For the other three instruments, around 70% of the respondents felt that the EC contribution was sufficient. In the case studies, that did not include CA nor SSA projects, the sufficiency of EC-funding at the project level was highest in the group of STREPs (85.7%), followed by the NoEs (75.0%) and the IPs (60.0%). 70% of the survey-respondents and 75% of the case study interviewees indicated that the contribution granted to their organisation was sufficient. The survey-results show that coordinating partners 9 are slightly more positive than the coordinators and the regular partners. These results are not confirmed by the case studies, where the regular partners (77.9%) seem to be more positive about the sufficiency at the level of the organisation than the coordinators (73.1%) and the coordinating partners (66.7%). Some of the coordinating partners explained that due to the fact that they did not receive a coordination budget, unlike the coordinators, the costs of participating exceeded the budget they received for the tasks they were responsible for. 9 We made a distinction between the real project coordinators, regular partners and an in between category of coordinating partners, i.e. partners that had coordinating responsibilities within the project consortium (like work package leaders) without being the overall coordinator of the project. 34

35 The survey-results show only small differences between the different types of organisations. The survey results differ not much across the different types of instruments either, except for the CAs. 43.6% of the respondents participating in CAs indicated that the share of the EC financial contribution granted to the organisation was not sufficient given the tasks assigned to their organisation. Respondents participating in instruments that are more targeted to strategic research projects, such as IP s, STREPs and NoEs, are generally quite satisfied with the level of EC-funding. This is confirmed by the case studies, where the group of interviewees participating in STREPs comes first (78.6%), but is closely followed by the group of IPs (73.7%) and NoEs (73.3%). The most frequently cited budgetary problems at the level of the projects, mentioned by the case study interviewees, are: Cuts in the budget during negotiation stage or during first stage of project; Unexpected costs, for example experiments that turned out to be more expensive; Too heavy administrative burdens and coordination costs; Too ambitious objectives, in the call or even at the programme level; The most frequently cited budgetary problems at the level of the organisation are: Too heavy administrative burdens and coordination costs; Problems with timeliness of EC payments; As a consequence, the following types of partners appear to be most susceptible to budgetary insufficiencies: Partners involved in the coordination, especially in larger consortia. The 7% coordination and management fee was too limited for several projects studied in the context of the case studies. Costs related to coordination and management are not proportionally, but exponentially increasing with size (in terms of number of partners). Therefore, management and coordination costs are especially burdensome in larger consortia (mostly IP and NoE-projects). Partners involved for only a minor task. Some of the administrative requirements have to be fulfilled by all partners. Costs related to these tasks are mainly fixed. For these partners that only receive a very small part of the budget, these costs often exceed the budget (income). Small and medium sized companies. 35

36 Box 3: Insufficiency of EC-contribution examples from the cases 10 Cuts in the budget Europrevall (IP): The project was cut at the negotiation stage from 17 to 14 millions. As a consequence of the cut, the coordinator never got the money needed to run the project. As a rough estimation, the partners probably contributed 2.5 to 3 millions of their own funds. One of the research tasks, the preparation of food for the diagnosis in food allergies, could not be executed. Then, it was decided to outsource this task to the private sector, that can prepare this for free. But it took 2.5 years to arrange this, which jeopardised the whole timing of the project. Another aspect badly financed were the tools needed for a big project to be managed such as the web tool. In addition the travel budget was underfinanced given the size of the project. The budget cut also prevented interesting aspects to be tackled. For example, the interesting part dealing with placenta sampling and identifying relevant immunological cells and antibody levels which drive immuno tolerance/ development of atopic diseases were explicitly cut out of the project during negotiation phase. Too heavy administrative and coordination costs Truefood (IP): The funding for the management activities was not sufficient, according to the interviewee in charge of these activities. The ceiling of 7% was considered not very adequate given the big size of the project. For instance, the partners have to present a final audit certificate. If one counts on a budget of to euro for such audit, it is a very big share of the management budget. Un -timeliness of EC-payments Helena (large STREP): Up till now, 1.5 years after the completion of the project, the participants are still waiting for the last share of the funding. One interviewee mentioned that for universities involved in many EU projects, this is a big problem, since prefunding is offered at the level of the university (not at the level of the faculty or the department). The reason for the delay is that the EC is checking some financial and administrative documents where there appear to be some small problems (e.g. one document was handed in in French instead of in English). On top of that the scientific officer responsible for the project changed several times which also caused delays. Too ambitious objectives Sabre (IP): The general problem with framework calls is that they try to please too large an audience, and the research topics are too broad. In writing a proposal, you must try to address every word in the call. In doing that, the money is sometimes spread too thinly. It would be better to make research topics narrower and deeper in the future. This will consequently lead to a more targeted allocation of EC research funding. We conclude that according to about 3 out of 4 survey respondents and case study interviewees the EC-funding was in general sufficient both at the project level as for their own organisation. Partners that are most susceptible to budgetary insufficiencies are partners involved in the coordination (especially in larger consortia), partners involved for only a minor task and SMEs. 5.3 Additional funding sources The following figure presents the distribution between the different sources that the survey-respondents called upon in order to meet the costs not covered by the EC financial contribution. Important to note is that the underlying survey question was only asked to private organisations. 10 A more extensive list of examples can be found in the background report. 36

37 Figure 5-2: Private funding sources - by type of (private) organisation 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% 6.3% 2.9% 7.0% 12.1% 5.4% 7.0% 30.9% 6.1% 14.7% 66.2% Private research organisation 85.9% SME 93.8% Large company 81.8% 79.9% Other private organisation Total Internal financial sources External private financial sources External public financial sources Source: IDEA Consult on the basis of survey results The costs not funded by the EC were mainly covered by internal financial sources (80%). For large companies and SMEs the share of internal sources is 15 to nearly 30%-points higher compared to the share of internal sources of private research organisations. Conversely, the share of external public financial sources is 4 to 5 times higher for private research organisations in comparison with the other types of organisations. For companies, it seems that external private financial sources purely addressed to research are difficult to obtain. These figures are perfectly in line with the responses we obtained in another study 11 (on the behavioural additionality of FP6) where a similar question was posed to the company participants but across all thematic priorities. These companies (n = 709) also mostly rely on internal private financial sources to cover up the total cost of their project (80% of the companies ticked this source). External private financial sources was ticked by 8% of the companies. The survey results are confirmed by the case studies, where the large majority of the private organisations (75.9%) mainly meet the uncovered costs by internal financial sources, followed by external public financial sources (17.2%). In some of the projects, insufficient funding was solved by reallocating the budget from the partner with a budgetary shortage to a partner with a budgetary surplus. Further, in some organisations (mostly higher education institutes or research organisations) uncovered costs were funded indirectly via extra efforts from staff. Thus, there was no direct allocation of new budget to the project, but they involved additional researchers, PhD-students or sometimes even masterstudents to fulfil the uncovered tasks. Box 4: Additional funding sources examples from the cases Helena (large STREP): Budgetary insufficiencies implied a lot more work for the researchers and personal commitment to the project. To compensate the tight budget, the interviewed partners applied for additional funding at the regional and national level, and complemented the budget with own resources and resources from companies outside the consortium. These external 11 IDEA Consult and Falck R. (2009). Assessing the behavioral additionality of the Sixth Framework Programme, Prepared for the EC, DG Research. 37

38 companies were willing to fund EU projects because of the high quality and high research standards they imply. QualityLowInputFood (QLIF) (IP): As a consequence of the budgetary cuts and the negative influence on the project results, a part of the project consortium decided to add some governmental money to improve the results. They employed some master-students to do the interviews and to enlarge the number of experiments. Isafruit (IP): In order to stretch out management funds, the coordinator introduced a project policy that audit funds could not be refunded for consortium partners. Each management committee member received a lump sum for his participation, but this did not cover their actual costs of participating in the committee when salary expenses are summarised. What role has private investment played in the projects especially that from small and medium enterprises? Before answering this question, we analyse how well represented companies and private research organisations are in Thematic Priority 5. Participation of private organisations in Priority 5 Figure 5-3: Share of participations and participants EC contribution - by type of institution of the participants Source: Idea Consult on the basis of the FP6 Dbase and updated information of project coordinators Figure 5-3 shows the distribution of the participations and the participations EC contribution over the participants type of institution. Higher education institutes represent more than a third of the total number of participations. In terms of EC contribution their share is with 44% even higher. The public research organisations are strongly represented as well with 24% (27%) in terms of participations (EC contribution). Private research organisations count for 13 % (16%) 12 in terms of participation (EC contribution). Companies are responsible for nearly a quarter of the number of participations, but only 10% if one takes into account their relative share of the EC contribution (see infra). This supports the observation made in the FP evaluation literature that the FP is becoming more academic in nature. Looking at the distribution of the participations by type of institution over the different instruments, we find that for the new instruments, companies count for approximately a quarter of the total number of participations. We can therefore conclude that the role of private organisations in the Priority-5 12 The remaining 4% (3%) participants are other public organisations. 38

39 programme is rather limited. Even in IPs, the representation of private entities is very low. Companies represent only 23% of the total number of participations in IPs. In terms of EC contribution, companies only count for 11%. Only 16% of all projects is coordinated by a company. For IPs, this is even only 9%. These seem to be low figures, since IPs in particular aimed at a more substantial industrial participation. Role of private organisations in the project Next, we report the results regarding the role of the different types of organisations in the project. Table 5 1: Type of organisation by role in the project COOR (n=100) PART- COOR (n=363) PART (n=289) Total (n=752) Higher Education Institute 33,0% 38,8% 33,6% 36,0% Research Organisations 46,0% 40,8% 42,6% 42,2% Companies 6,0% 14,6% 15,2% 13,7% Other 15,0% 5,8% 8,7% 8,1% Total 100% 100% 100% 100% Source: IDEA Consult on the basis of the EC database and the survey Research organisations are most represented in all three categories, especially in the group of coordinators (with 46%). One third of the coordinators are respondents from higher education institutes. Companies represent only 6% of the coordinators. We conclude that the private participants covered the costs not funded by the EC mainly by internal financial sources (80%). In some of the projects, insufficient funding was solved by reallocating the budget from the partner with a budgetary shortage to a partner with a budgetary surplus. Further, in some organisations (mostly higher education institutes or research organisations) uncovered costs were funded indirectly via extra efforts from staff. Private investment from the companies is albeit rather limited because companies are responsible for nearly a quarter of the number of participations, but only 10% if one takes into account their relative share of the EC contribution. This difference can only partially be explained by different cost rules. Companies also represent only 6% of the coordinators. 5.4 Management Efficiency of the project management Overall, 91% of all the survey-respondents agree that the project has been or is being managed and coordinated efficiently. With 86%, this share is somewhat lower for the partners that are not involved in the project management. This can 39

40 be explained by the fact that the coordinators and the coordinating partners are probably less inclined to criticize their own decisions. Further, there are no great differences between the different types of organisations. The case studies confirm this positive assessment, by the participants, of the overall project management efficiency, with 93% of the cases that were efficiently managed, according to the interviewees. The results by instrument show that this share is highest for the STREPs (all projects were considered to be managed efficiently), followed by the IPs (93.3%) and the NoEs (87.5%). The fact that the results are most positive for the STREPs may be explained by the size of the consortium that is typically smaller under this instrument. The following issues, which are under the control of the consortium, have an enhancing influence on the efficiency of the management: - A stable coordination team: In one of the projects, the coordination team changed several times throughout the project, which was problematic for the progress of the project. - Previous coordination experience, preferably in similar types of projects: The coordination of large FP-projects is a learning process. Coordinating experience in a previous FP (or another kind of EU or international) project is an important advantage. - High transparency and information-sharing: most of the interviewees that are/were not involved in the coordinating team were satisfied with the transparency of information provided by the coordinator. - Good communication: The daily communication between the coordinator/coordinating team and the other partners mostly takes place by and telephone. In addition, most projects have one or a few annual meetings that all partners should attend and more frequent meetings at the level of the work packages or sub projects. - A substantial involvement of all partners in the consortium: In some consortia, some of the partners were only involved to cover the small remaining uncovered topics in the call in order to win the bid. The activity level of these partners in relation to the whole project was insufficient, resulting in an administrative burden (largely a fixed cost) that exceeded their income level. Influence of EC rules and practices In general, the survey-respondents don t support the idea that EC rules and practices enhance the efficiency of the project management, but rather have a neutral impact or even the opposite (with one exception, all means are between 2 = hampered to a limited extent and 3 = no influence ). Most hampering are the financial rules and reporting requirements and the time required by the EC to implement contractually agreed payments (with a mean of 2.31 and 2.39 respectively). Rules considered to have a more or less neutral impact on the project management efficiency are the scientific reporting requirements (mean 3.08) and the share of the budget explicitly allocated to management and coordination tasks (mean 2.95). Overall, these results confirm the need for simplification of the Framework Programmes. The results of the case studies are in line with the survey results. The financial rules and reporting requirements and the time required by the EC to implement contractually agreed payments (with a mean of 1.95 and 2.05 respectively) are 40

41 also considered to be most hampering. One of the coordinators specifically mentioned serious inconsistencies in financial rules and reporting requirements. Rules considered to have a more or less neutral impact on the project management efficiency are the scientific reporting requirements (mean 3.05) and the duration of the contract negotiations with the EC (mean 3.00). If we look at the results by role, we notice that the scores of the coordinators are somewhat lower for the different types of reporting requirements (scientific, management and financial) compared to the scores of the coordinating partners. This can probably be explained by the higher responsibilities of the coordinators with respect to the administrative tasks of the project. Conversely, the coordinators were slightly more satisfied with the share of the budget explicitly allocated to management and coordination tasks, which is confirmed by the case studies. From the case studies it became also clear that most of the EC rules and practices are considered to be more hampering in the IPs and the NoEs, compared to the STREPs. This can be be explained by the fact that IPs and NoEs are, on average, larger in terms of consortium size. Some important remarks from the cases: The contract negotiations, communication and cooperation with the EC were, in general, not perceived as problematic. However, some of the interviewees mentioned that the feedback of the EC on evaluation reports was rather (even too) late. Finally, the EC presence on the different meetings and conferences could have been better in some of the projects. The flexibility in terms of transfer and changes in the budget between the partners can have a positive effect on the effectiveness of the management. Especially hampering are the rules and practices related to finance. Besides the EC-financial rules and reporting requirements and the time required by the EC to implement the payments, also the budget cuts during the negotiations and the execution stage are considered to be very problematic in some of the projects. With regard to the reporting requirements: o o o Fewer, shorter and more relevant intermediate reports would have been better: reports containing milestones instead of real deliverables. The same holds true for audits. While they are a very useful controlinstrument, they are very demanding in terms of time and efforts. Rules, especially financial rules, and the way how you should report were not always clear. More knowledge initially about the management and financial guidelines would have been helpful. There are many guidelines available though which seems to imply that these are not clear enough or do not reach their target group. The scientific reporting requirements were more manageable. The administrative requirements are for most participants a learning process for which you have to invest time and effort. In addition, the administrative procedures have been reported to change frequently, thereby increasing the time that is needed to fulfil all administrative tasks. o Despite all these disadvantages and problems, the intermediate deliverables can also be very useful, since they help to redefine and focus the work plan for the next project stages. 41

42 The instability of the EC-team responsible for the project seriously hampered the management efficiency. In several projects, the scientific and/or financial officers in charge changed a few times during the project. This has some harmful consequences, since: o o o It has a negative influence on the effectiveness of collaboration, Expertise, quality and view on certain issues of these scientific and financial officers can strongly differ, The coordinator has to spend much time and energy on renewing contact. Changes in FP7 According to the interviewees, the EC management and financial reporting requirements have not really been simplified. Especially the day-by-day time reporting is considered to be very burdensome. On the other hand, improvements are made concerning the methods of cost calculation and the provision on the audit certificate with a higher threshold level. The elimination of the cost models, and thus of the obligation to select a specific cost model reduces rigidity and the overall workload. Further, the ceiling of the 7% management fee has disappeared, which will make it easier to cover all managerial costs. Overall, 91% of all the survey-respondents agree that the project has been or is being managed and coordinated efficiently. The case studies confirm this positive assessment, by the participants, of the overall project management efficiency. The following issues, which are under the control of the consortium, have an enhancing influence on the efficiency of the management: a stable coordination team, previous coordination experience, a high transparency and informationsharing, good communication and a substantial involvement of all partners in the consortium. In general, the survey-respondents as well as the case study interviewees don t support the idea that EC rules and practices enhance the efficiency of the project management, but rather have a neutral impact or even the opposite. Most hampering are the financial rules and reporting requirements and the time required by the EC to implement contractually agreed payments. These results confirm the need for further simplification of the Framework Programmes, which has been taken up, at least partially, under FP7. 42

43 6 PARTNERSHIPS 6.1 Introduction The partnerships in the project consortia are relevant for the evaluation, as this is a foundation that can support the assessment of the degree to which the projects have achieved the objectives. In this regard a number of evaluation questions were posed. In addition, we propose some additional issues that are relevant to look into in order to get a better understanding of the project partnerships. In particular this relates to the prior collaboration of the partners as well as the continued collaboration of the partners. Prior collaboration is important as this can indicate the background for these partnerships and the continued collaboration patterns will indicate us e.g. to which extent the EC projects are sustained via networking, continued research etc. Finally, we also address the participation of SMEs in further detail. The following evaluation questions will be answered in the section of partnerships, in this order and grouping: 43

44 EQ 7: To which extent did the project partners collaborate before the project? EQ 8: To which extent did the project partners continue their collaboration after the end of the project? EQ 9: Was the project size and participation optimal especially with respect to the new instruments? EQ 10: To which extent has international cooperation with third countries been implemented in the projects and via which instruments? EQ 11: Has international co-operation through the inclusion of third countries in the projects been of benefit to either the EU and/or the third party? EQ 12: What has been the role of new member states in the projects? EQ 13: Were the third country participants of high international standard? EQ 14: To which extent did SMEs participate in the projects and via which instruments? EQ 15: What was the role of SMEs participating and what characterises their participation? EQ 16: What role has gender played in the overall projects? The answers to these evaluation questions are discussed in the following sections: Project collaboration Project size and partner mix Cooperation between old and new member states and third countries SME participation Gender balance 6.2 Project collaboration The evaluation questions that are answered under this section are focused on the collaboration prior to the projects and the continued collaboration after the projects Prior collaboration When describing the profile characteristics in the background report, we already mentioned the high level of continued participation of the individual respondents across the different FPs. In addition and (partly) resulting from these continued participations, FPs are characterized by the usage and expansion of existing networks. This is further confirmed by the survey results. About 75% of all respondents from the survey have collaborated with some of the project partners on a research project before, and nearly half of them within the context of an FPproject. 44

45 The following box presents a few examples from the case studies on how the partnerships were set up. A general observation is that project partnerships nearly almost emerge from existing relations. Two things in particular are demonstrated by the cases: (1) project consortia are often build on existing relations, leaving the core of the consortium to be a group of already established and experienced partners, and (2) consortia often extend to the network of the network and sometimes through formal calls, which opens up to new partners. Box 5: Partnership set-up - examples from the cases Before the start of FP6, the coordinator of BIOCOP (IP) handed in an expression of interest regarding the subject of BIOCOP. Eventually, a part of that expression of interest appeared in a call under the 5 th Thematic Priority of FP6. Following to that, the coordinator established a core group of people from organisations he knew from collaboration either on the national or the European level. After the focus of the project was established by this core group, they identified the gaps and launched an expression of interest for possible participants in the consortium. One of the interviewees mentioned that from those partners with whom there was prior collaboration, he knew they could be trusted and deliver qualitative results. Working with new partners on the other hand is a conscious choice because of the aim to expand the network. The PATHOGENCOMBAT (IP) project was initiated via an expression of interest that a group of the partners prepared in The expression of interest was included in the call for tenders, and therefore the group decided to submit a full application. A group of seven partners knew each other very well from previous collaboration, and this group prepared the expression of interest. The remaining 32 partners were involved via this group of seven partners. In this manner, all partners knew some of the PATHOGENCOMBAT partners, but most were also introduced to new partners. A precondition for the successful application was the international network that was gathered. ACE-ART (STREP) started as a response to a need for drug resistance profiles concerning the use of certain bacteria in animal feed. Although not formally involved in the expression of interest, several scientists belonging to the European Food Safety Authority (EFSA) were involved in the predevelopment phase of the project. Initially, there was a call for an expression of interest to which 31 organizations replied. After assessment of the quality of the applications, 14 organizations actually started participation in the ACE-ART project. All interviewed organizations have collaborated with one or more of the partners prior to the beginning of the project. However, these networks do not appear to be limited to a closed circle, i.e. each interviewee referred to different partners. Several interviewees also specifically mentioned their awareness of the existence of the other partners due to their presence in scientific literature. The cases further indicate that there are a number of benefits from building on previous experiences from collaboration: Firstly, it is a demanding task to be coordinator and so it is valuable both to have experience from participating in similar projects before and to know a number of the participants beforehand. Secondly, many of the case studies reveal that collaboration is encouraged by the confidence between the participants and that building up such confidence takes time. Thirdly, the case studies show that it takes some time to build up collaboration structures. Since the projects are limited in time, it is often beneficial to have a good starting point from a prior collaboration. The case studies further indicated that SMEs have less collaboration experience from before and also competitiveness concerns might refrain companies from disclosing all their knowledge to other participants; combined this can imply that the SMEs are not always fully integrated into the projects. The tendency to extensive use of existing networks could hamper the entry of new partners. Although, if we look at the results of continued collaboration (see below), we notice that within Priority 5, the scope for new partners including industry seems to be large enough. 45

46 6.2.2 Continued collaboration In the following paragraphs, we discuss the results concerning continued collaboration, i.e. collaboration with some of the project partners after the completion of the project. We present the results for the respondents that participated in the projects that have been finished already 13. The survey data show that in total, 28% of the respondents indicated that up to now no collaboration has been continued with any of the partners. In cases where collaboration has been continued, it has mainly occurred outside the context of EU FP (36% only outside the EU FP context, and 16% in combination with collaboration under EU FP). The limited attractiveness of EU FP for continued collaboration can be explained by the fact that FP projects are targeted at specific subjects. If the specific subject insufficiently corresponds to one of the research areas that are incorporated in the subsequent programme, then collaboration is more likely to continue elsewhere, such as under national research programmes, international programmes other than FPs, privately funded research projects and so on. Companies and uncategorised organisations ( others ) show the largest shares of no continued collaboration (38% and 44% respectively). Of the 62% of the companies that did continue collaboration more than half did so outside the context of EU FP. This is not necessarily a discouraging result as it is expected that part of the path from basic research results via applied research results to improved food products or production processes will take place outside a research platform - i.e. on a market platform. This market platform might involve several participants from the project, both companies and other types of participants. Collaboration does not only continue with existing partners. Even though FPs are characterized by the use of existing networks, there is still enough scope for new partners in FP6 TP5, with 59% of the respondents indicating that they continued collaboration with some new partners. If we look at the type of new partners, we notice that these are, overall and in absolute numbers, mostly universities and research organizations from abroad, followed by domestic universities and research organizations. The relative share 14 of collaborations with industry is however higher which indicates that industry is more involved after the end of the project. From the survey it is evident that NOEs have a higher level of continued collaboration compared to other instruments. Only 8% of the NOE participants did not continue collaboration compared to an average of 28%. This also shows as NOEs having a slightly higher level of continued collaboration in the case studies (both ongoing and finished). More than nine out of ten NOEs continue collaboration and eight out of ten continue under the EU FP. The reason for this result is that the NOEs have had particular requirements to the durability of the established networks, and most of them will also continue in some form of legal entity, primarily as associations registered in a given host country. These entities The survey results of the ongoing projects are not included here, since for these projects the situation of continued collaboration is a hypothetical one. However, the results for these did show highly positive expectations towards continued collaboration compared to the actual results for the finished projects (more information on these expectations is available in the Background Report). Taking into account the overall industrial participation in the survey which was at about twenty one percent. 46

47 are lighter versions of the previous NOE structures with a slimmer management setup, possibly fewer partners and fewer activities. Overall, there is evidence of extensive collaboration both before and after the FP6 TP5 projects, also with the new partners. Prior collaboration has displayed a number of benefits to increase the success of the projects, and continued collaboration seems to include both existing and new partners across types and countries (via FPs). 6.3 Project size and composition The evaluation questions that are answered under this section are focused on whether the project size and composition was optimal, especially with respect to the new instruments Consortium size The number of contractors in the projects funded under TP5 is large compared to the other TPs, except for NoEs (see Table 6 1). The IPs and the CAs funded under TP5 are much bigger in terms of the number of contractors compared to the other TPs. Overall, TP5 projects involved more than 17 contractors whereas an average project (taken across all Thematic Priorities) took a little less than 14 contractors aboard. Table 6 1: Average number of contractors - by thematic priority and instrument Thematic Priority IP NoE STREPS CA SSA Total 1 Life sciences (n=601) IST (n=1090) Nanotech (n=445) Aeronautics & Space (n=241) Food Quality & Safety (n=181) Sustainable development (n=669) Citizens and Governance (n=146) All Thematic Priorities (n = 3373) Source: Idea Consult on the basis of the FP6 Dbase and updated information from project coordinators (n refers to the number of projects funded) The majority of the respondents in the survey (63%) agreed that the number of partners in the consortium was/is optimal. If not optimal, respondents indicate there were/are too many partners (35%). Only 2.2% responded there were/are not enough partners involved. The more coordinating responsibilities, the higher the percentage of respondents that considers the consortium size as optimal (75% of the coordinators opposite to 66% of the partners involved in the project management and 55% of the other partners). A first possible explanation is the assumption that coordinators and partners involved in project management are probably better equipped to deal with large project consortia. Another possible explanation is that coordinators are less 47

48 inclined to admit the sub optimality of the consortium size, because they were more strongly involved in the composition of the consortium. Table 6 2: Optimal consortium size - by instrument (a) IP (n=249) NOE (n=102) STREPS (n=195) CA (n=40) SSA (n=158) Total (n=744) n Not enough partners 0.4% 2.1% 2.6% 0.0% 5.1% 2.2% Optimal number of partners 34.5% 62.8% 83.1% 57.9% 82.1% 62.7% Too many partners 65.1% 35.1% 14.3% 42.1% 12.8% 35.1% Total 100% 100% 100% 100% 100% 100% (a) Was the size of the project consortium optimal (in terms of the number of partners)? Source: IDEA Consult on the basis of survey results The analysis by instrument shows that more than 80% of the respondents participating in STREPs considers the consortium size as optimal, opposite to the respondents participating in IPs, where this is only 35%. Given the fact that IPs are in general very large projects in terms of numbers of partners, this result is not very surprising and in line with the comparison between the thematic priorities. On the contrary, only 35% of the respondents participating in NOEs (on average also many-partner-projects), indicate a surplus of partners. The above tendency from the survey for a result towards too many partners is more dominant in the case studies 15. Almost half of the cases reckoned that too many partners were involved. Just like in the survey, there are large differences by instrument. Almost seven out of ten IPs are/were too large according to the interviewees. Opinions on the size of the NOEs were more scattered, as ¼ of the interviewees found there were not enough partners and ¼ thought there were too many. Comparatively, the size of the STREPs was most optimal, in seven out of ten cases. Some consequences of projects being too large: There are projects in particular IPs - where some participants find the research topics or their interests too diverse, i.e. two or more separate projects would have been more appropriate. There are also projects where participants find their budget allocation too small because of too many participants. In particular for the NOEs, the interviewees often mentioned the lack of funding for research as the main issue. In some projects the size of the consortium increased the administrative issues including reporting requirements to an extent that was challenging for the coordinator. The following box displays case examples on size. 15 Note that these only include IPs, NoEs and STREPS. 48

49 Box 6: Observations related to size examples from case studies The HEALTHGRAIN (IP) consortium was perceived as being too large by the interviewed participants (44 partners, an industrial platform of 61 companies, a network with 23 experts and a consumer group of 11 experts). The reason for this was the administrative burden generated by the high number of partners. One partner pointed out that if the consortium had only had been around thirty partners, more funding could have been allocated to the best researchers. The result would have been less scattered research, which from his point of view prevented HEALTHGRAIN from progressing further in terms of research outputs. Contrary to this point of view, an advantage of being big is that many new links are created between different disciplines and research areas. Overall, another disadvantage of a big project is that it can be difficult to change the research direction of the project once it is started, and much flexibility is lost. From the outset it was a major objective of NEUROPRION (NOE) to gather all relevant research teams and therefore the consortium became quite big. One partner mentioned that it was because of this objective that the consortium became too big. It resulted in the project becoming quite unmanageable and the overall objectives became unclear to the participants whos interests might have differed to some extent. From this partner s point of view, the same results could have been achieved without the high number of meetings and extensive networking. One stated that a more strategic selection of partners would have been beneficial, and that four out of ten could have been merely associated partners. Despite ISAFRUIT (IP) being a large project involving 60 partners, the interviewees agreed that the size of the project was appropriate. Due to its magnitude, it was possible to gather many disciplines, and it was suitable for European-wide networking. Such a large project can be challenging to handle, but the coordination set-up with a management group and several pillars of work packages dedicated to certain topics functioned well. One disadvantage of the magnitude of the project was the comprehensive reporting requirements constituting several thousand pages. This was also commented by the external reviewer. Another weak point was that communication between the different work packages mainly took place at the WP management level. Advantages of having a larger project are that the consortium gathers critical size - that is to gather a sufficient number of partners with knowledge and within a particular research topic. In addition there is potential to reach economies of scale among similar partners. Disadvantages of being large is that communication among all partners can be challenging, e.g. there are several examples where partners are not aware what is going on in other work packages. In contrast, where a goal of a consortium is to influence the policy context via the research findings, there is a risk of a too small consortium "not to be able to speak with a load voice". Looking closer at the optimal consortium size of IPs, we find that more specifically in terms of partners of a specific type of organization, the high percentage of too many partners is explained by too many higher education institutes and research organizations. Conversely, there are not enough companies and Industrial Associations and Groupings (IAGs) involved in the project consortia. The same is evident in as much as half of the cases that show that in particular large companies are not well represented, but SMEs and industrial associations could be better represented as well Partner mix This section discusses the optimal consortia in terms of the mix of partners. Looking at which types of organizations are considered to be under- or overrepresented in the project consortia, from the point of view of each of the different types of organizations, the survey data show that for a substantial part of the respondents large companies, and to a lesser extent SMEs were indicated as being under-represented in the consortia. 49

50 This last result confirms one of the basic principles of the Framework Programmes, that is to say the importance of cross sectoral collaboration between the more academic and basic research conducted by higher education institutes and research organizations on the one hand and the more applied research of companies on the other hand. In cases where there are too many partners involved in the project consortium, this is mainly due to an overrepresentation of higher education institutes and research organizations. The appropriateness of the mix of participants in the different projects was a central issue addressed in the case studies. While some participants mainly participate for their private interest, there are many that have seen the synergy or economies of scale benefits of collaboration and so put much emphasis on having the right mix of partners. Furthermore, many projects had to overcome the challenges of bringing together different research disciplines or different types of partners. Partners see synergy benefits from looking at food quality and safety issues from different angles e.g. different research methods or focuses on different food products, and in this context to have a balanced decision-making process (mostly lacking participants with practical implementation experience). There is as mentioned above a need for different types of partners to ensure a transfer of research results to the market place or to the legislative process. The following box display examples of the challenges from the particular mix of partners. 50

51 Box 7: Examples from the cases on multidisciplinary collaboration In TRUEFOOD, the very good mix between science and industry was, according to the interviewees, one of the strengths of the project. The scientific partners could benefit from the practical experience from the industry floor. This mix was very well balanced: there were 20 industrial partners and 23 scientific partners. There was nevertheless the challenge to manage this diversity. Partners needed to learn to speak the same language which was not the case from the beginning. But they learned a lot from each other and got to understand each other. For QUALITYLOWINPUTFOOD, cooperation among different disciplines improved the quality of the results. Especially the methodology sharing and the sharing of research results was quite high across the different disciplines (socio-logical, economic, scientific). For example, the exchange of the results of the different national research trials and the sharing of infrastructure to make better use of the facilities was very cost efficient and had a positive impact on the quality of cooperation. One of the partners mentioned that TRACEBACK was the first project in which they were really being challenged to cooperate with people from different disciplines (i.e. ICT, nanotechnology and food research) as they had to overcome differences between the technical languages. To give an example, the coordinator explained that the word evaluation has a totally different semantic meaning in nanotechnology than in the food sector. Consequently, the partners spent a lot of time in meetings and on the phone to learn each other s language and be able to cooperate successfully. In that sense, the pre-pilot, organized to help the partners recognize the needed input, was indicated by the interviewee as a helpful tool in streamlining languages. PARASOL (STREP) is an example of a project with commercial success that received the highest assessment on all parameters by the food research expert who reviewed the case. The main idea of PARASOL was to replace the current practise of treatment by targeted selective treatment (TST). The tools for TST were developed, refined, standardised and validated in the project an approach that addressed the most important key research issues. The number of papers and presentations was impressive, leaving the impression that there has been a strategy and focus on publications all through the project, and results published in high impact journals. Outputs of the project include guidelines and recommendations on the tools. Particularly, the assessment highlighted that four commercial products (FLOTAC, Svanovir, Grassmaster II, FecPak) were developed during the project, which was excellent within the short timeframe of the project. The major reason was perceived to be the integration with private companies allowing ideas to be disseminated and taken further by the relevant persons. In addition to specific products, the commercial success also is that developed TT/TST strategies may be widely used for the benefits of farmers, which will also contribute to economic benefit of the project. More products could be developed in the future, e.g. further development of molecular methods to determine AR, pen-site devices for use of farmers and results of P-glycoprotein would be basis for development of improved drugs. The expert did question whether more focus should have been on demonstrating the benefits of these tools combined with sociologic studies of implementation issues. However, compared to other initiatives in the field, in conclusion PARASOL was the best. Similar to the analysis of optimal consortia in relation to the type of partners, we conducted the same analysis in relation to optimal country representation. From this analysis we can conclude that the majority of the respondents from the different country groups consider the country distribution of the consortium as optimal. Sub optimality is mainly due to the under representation of a certain country group in the consortium, mostly candidate countries and third countries. An exception here is the group of EU15 partners, who are considered to be overrepresented in the consortia by 14% of the respondents. The case studies have shed further light on the importance of securing a good country mix among the participants. There are food quality and safety issues that differ between geographical regions such as: differences in crops grown in particular countries (including GMO versus non-gmo), structures of food chains, differences in the use of fertilisers or pesticides or differences in weather conditions e.g. between northern and southern Europe. The benefit of comprising such country-specific differences is the ability to analyse how much such differences affect food quality and safety. 51

52 What is striking about the survey results is that both in regards to partner mix and country mix sub optimality of a specific type of partners (representing a particular type of organizations or organizations from a particular geographic area) is mainly explained by a shortage of this type of partners. On the contrary, the results of the general question regarding consortium size optimality show that sub optimality is mainly due to a surplus of partners. This allows us to conclude that a project consortium might be optimal, or even too large in terms of numbers of partners, but that at the same time, a specific type of partners or countries might be lacking in the consortium. Overall, the project size has been optimal in most cases. When not, then projects consortia have been too large. This was primarily the case in the integrated projects. The statistics comparing the average number of contractors across thematic area indeed show that the consortia for the IPs under TP5 were by far the largest. The case study results show that there are negative consequences when projects are too large. With respect to the partner mix, a substantial part of the survey respondents indicated that large companies, and to a lesser extent SMEs are underrepresented in the consortia. In cases where there are too many partners involved in the project consortium, this is mainly due to an overrepresentation of higher education institutes and research organizations. In terms of country representation we conclude that the majority of the respondents from the different country groups consider the country distribution of the consortia as optimal. The case studies have shed further light on the importance of securing a good country mix among the participants, for instance because there are food quality and safety issues that differ between geographical regions. The benefit of comprising such country-specific differences is the ability to analyse how much such differences affect food quality and safety. 6.4 Cooperation between old and new member states and third countries The evaluation questions that are answered under this section are focused on the extent of international cooperation, the benefit of including third countries as well as their international standard, and the role of New Member States in the projects The extent and importance of international cooperation The food projects generate cross border links, compared to domestic links, which demonstrate the wide extent of international cooperation in the FP6 TP5 projects 16. More than 95% of the survey respondents agree that transnational collaboration is at least rather important for the achievement of the project objectives (with 16 See background report for the detailed matrix of domestic and cross-country cooperation. 52

53 72.4% finding it even crucial). Results from the cases support this result, with about eight out of ten cases being positive on international cooperation. The respondents from EU10 countries are the most convinced of the importance of international cooperation (mean of 2.81), while respondents from the candidate and associate countries have indicated the lowest scores (mean of 2.59 and 2.61 respectively). This last result was quite surprising, since we expected that they are even more dependent on transnational cooperation. However, from the case studies it is clear that also most of the interviewees from candidate and associate countries regard the EC FP programme to have large value added compared to national or regional programmes. So the survey and case study results strongly support the rationale behind the Framework Programmes: the importance of cross national research and collaboration is clearly confirmed New Member States Overall, New Member States amount to 313 participants equaling about 10 % of the total number of participants. Comparatively NMS receive 5.2 % of the EC contribution. In 3 % (5 projects) of the FP6 TP5 projects, the coordinator was located in a NMS. Most NMS cross country linkages were to partners from the OMS, in addition a few were within NMS countries, candidate countries and associate countries. Table 6 3 shows that the share of old member states participants (OMS) for TP5 is below average compared to the other TPs, in favour of a share above average of new member states (NMS), candidate countries (CC) and third countries (TC). We further noted that the share of the participants from candidate countries and from third countries (in particular) increased over time (see background report). Table 6 3: Share of participants- by thematic priority and country group (a) Thematic Priority OMS NMS CC AC TC Total 1 Life sciences (n=601) IST (n=1090) Nanotech (n=445) Aeronautics & Space (n=241) Food Quality & Safety (n=181) Sustainable development (n=669) Citizens and Governance (n=146) All Thematic Priorities (n = 3373) (a) OMS: Old Member States; NMS: New Member States; CC: Candidate Countries; AC: Associated Countries; TC: Third Countries Source: Idea Consult on the basis of the FP6 Dbase and updated information from project coordinators (n refers to the number of projects funded) With respect to the New Member States (EU10), the case studies results indicate that in more than half of the cases there are not enough NMS involved in the survey 29% of the respondents indicated that NMS were not sufficiently represented, while this was 43% for candidate countries. This relatively low result can be explained by the finding that only in half of the cases NMS inclusion was considered important, for which reason there also have not been made initiatives to increase the share of NMS, apart from the initiatives to adhere to EC requirements. 53

54 In any case, it is important to emphasize that participants mainly are included because of their competence and not their country of origin. Several of the interviewed participants have expressed that there are often different levels of collaboration among various partners. Some state that participants from the New Member States in general have more limited research capacity, while others are positive and indicate that the contribution of the NMS was equivalent to the OMS. The following box presents a number of cases where participants from NMS lacked competences, knowledge or infrastructure, which explains their lower participation in the projects. Other examples illustrate that positive collaborations took place. 54

55 Box 8: Representation of the New Member States examples from the cases Only one new member state, Slovenia, was represented in EADGENE (NOE). The other partners observed no differences in knowledge level and mentioned that the particular researcher involved was educated in Germany. However, the point of departure for the research was different, as there was less money available as well as limited equipment to undertake research activities. The mix of countries was wide in NEUROPRION (NOE), as 20 countries were represented. However, only one new Member State, Poland, was represented in the network. The challenge with including NMS was that activities within this particular research field was minor in Eastern Europe and therefore the number of research teams from NMS marginal. The mix of countries in BIOCOP (IP) was quite diverse, since for FP6 the focus was far less on the geographic scope of the consortium set-up. No partner was selected on the basis of its geographic situation but only on scientific performance. Since BIOCOP is a very high tech project, only high tech participants were included which implied only a small representation of New Member States. In ACE-ART (STREP), there was an overrepresentation of participants from the old member states. The only two exceptions were research institutes from Norway and Poland. Other interested parties from new member states or accession countries were not considered for final participation due to a perceived inability to participate and add to the research, indicating that adopting a better country mix would have been detrimental to the project s outcomes. For QLIF (IP), it would have been better to have more partners involved from Eastern Europe, because of the European enlargement. This was however not feasible in QLIF. Many researchers in the field come from Western Europe. Unfortunately, their view and their knowledge about consumers in the Eastern part of Europe was limited. The absence of a decent number of Eastern European partners made knowledge exchange complicated. Unfortunately, New Member States are still lagging behind in terms of funding and knowledge of languages. Therefore, the inclusion of NMS-partners is rather costly (e.g. consumer surveys have to be translated) and there is always a risk that after the project has finished, they will not be able to continue research at the same level. In EURRECA, there were only three different NMS involved. In order to have better and geographically wider spread results, it would have been better if more different NMS were involved. However, most of the NMS are rather small countries, and several of these countries are not targeted at the research that is needed for EURRECA. This made it difficult to find the right scientists and institutions in the other NMS. In addition, these countries are not equivalent to the OMS yet. There is a gap between the old member states and the new member states with respect to infrastructure, access to databases and existing knowledge. Especially the gap in know-how and knowledge is problematic. Maybe this gap can be reduced by additional funding on the one hand and by a decrease in the existing language gaps. Next, TRACEBACK (IP) had only two partners from new member states, both from Poland. One of the Polish partners also acted as a work package leader in the project. None of the interviewees mentioned any specific differences between new and old member states influencing cooperation or project outcomes. As one interviewee mentioned, I think all of them performed their job perfectly. The only New Member State participant in HELENA (STREP) was from Hungary. They contributed a major part to one of the WPs and according to the interviewee, their contribution was equivalent to the other participants from the Old Member States. Although in general, New Member States are still lagging behind from a scientific point of view, this was not at all the case for the Hungarian partner involved. An interviewee from the SABRE project sums up the point: The inclusion of partners from the New Member States can be very valuable. However, these partners sometimes lack the required infrastructure and laboratories, and also the right skills and expertise. Partners from NMS are often followers instead of leaders, not contributing to cutting edge science. Structural funding, such as the Marie Curie Fellowships are probably better than FP-projects to bridge this gap. 55

56 6.4.3 Third countries Overall Third country partners amount to 526 participants equaling 16.7% of the total number of participants. Third countries receive 9.5% of the total EC contribution. In six projects (3 % of the projects) a third country was coordinating. From the survey 64 % of the respondents found that the optimal number of third country partners was involved, while 30% found that not enough third country partners were involved. Table 6 4: Importance third countries - by type of organisation (a) Higher Education Institutes (n=208) Research organisations (n=245) Companies (n=73) Other (n=41) Total (n=567) N (3) Crucial 19.8% 22.6% 18.6% 24.3% 21.2% (2) Rather important 45.5% 40.7% 40.7% 37.8% 42.2% (1) Rather unimportant 34.7% 36.7% 40.7% 37.8% 36.6% Total 100% 100% 100% 100% 100% Mean (a) How important was the involvement of Third Countries* in the project for the achievement of the project objectives? Means with 3 = crucial; 2 = rather important; 1 = rather unimportant. Source: IDEA Consult on the basis of survey results Compared to the importance of transnational cooperation (overall average of 2.69), the importance of the involvement of third countries is less pronounced (overall average of 1.85). Not entirely surprising, the survey results show that the involvement of third countries is especially considered important by the respondents from third countries themselves (mean of 2.43, compared to for respondents from other countries). According to the interviewees in the case studies the benefits to the third country participants include the possibilities to improve skills, visibility, and the opportunity to train younger scientists. As the examples in the following box show, third country participation is in some cases quit important to the other participants which is somewhat contrary to the general results on importance from the survey (these examples are potentially outliers in the survey answers). Box 9: Third country participants One organisation from New Zealand and one in the USA are partners to ISAFRUIT (IP). The third country partners mainly participate due to their specific skills that they wished to connect to the project. In the two cases, international partners were selected due to their positions as world leaders in their respective fields. The partners have contributed to meetings and discussions drawing on their respective expertise. Australia participated as a third country in the PATHOGENCOMBAT (IP) project. They led WP 14 concerning Modelling microbial population behaviour throughout the food chain. The Australian participation was highly important, as input from this partner was unique to Australia, meaning that the company introduced new knowledge to the European partners that would otherwise not have been accessible. In general, the project largely focused on third countries. Trainees from Africa, Brazil, Mexico and several other countries went on exchange trips to the European institutions, and a seminar was arranged in Burkina Faso, where 11 African countries were represented. In addition, the US was represented in the management group. Iceland and Norway also participated in SEAFOODPLUS (IP) and based on extensive activities of 56

57 these countries in the sector, their participation was important. Especially Norway was singled out as a major contributor (stated by the Icelandic partner). Canada was also involved and provided contributions of good quality. Two third countries were involved in the BIOCOP (IP) project, Canada and Thailand, which were both important for the dissemination of the results and thus the reach of the objectives. TRACEBACK (IP) included two partners from third countries (i.e. a Turkish university and an Egyptian consultancy firm). These organizations played a relatively important role as organizers of the regional industrial platforms in their respective countries. The specific importance of having partners from these countries involved was related to substantial differences in the food chains compared to European producer markets. To give an example, one of the interviewees mentioned that setting up a number of ICT requirements for Egyptians was a completely new experience. Therefore, not having input in the design stage would reduce implementation opportunities and make the usability of the system poorer. Consequently, the input of these third countries should not be discussed along the lines of quality, although there were no complaints, but in light of a differentiation brought to the project. In SABRE, third country participants were involved for their expertise and to do the necessary research to reach the project objectives, not because of their geographic location. That was only an additional trump. The quality of these third country partners was considered to be good. The involvement of the third country partners in PARASOL, especially the two partners from South Africa, appeared to be very useful in order to reach the objectives. The major reason to include the South African partners was because in this country the problem of anthelmintic resistance is much more advanced. The PARASOL-team could take advantage of this practical experience that the South African colleagues gathered in the past. That particular knowledge and expertise was very important for the implementation of new guidelines in Europe. In addition, also the different climate conditions in the different regions were an advantage to see different results of the test. It was a challenge to harmonize these results of the different regions, but the project succeeded in achieving this harmonization. Only one Third Country was involved in the EURRECA consortium, but its involvement was very important. The organisation belonged to the scientific leaders in the research field. This was the main reason why they were asked to be part of the consortium. From these examples mainly two benefits arise to the additional European partners, namely (1) access to data, material, and knowledge particular to the third country due to weather conditions or experience, and (2) access to world wide leading scientists that contribute with new knowledge that is not available within the EU. The second point also indicates that third country participants typically were of high international standard. Overall, the international cooperation has been extensive (measured in cross border linkages), and importance of this type of cooperation was very high. New Member States were with about 10% of the participants represented well above average under TP5 compared to the other TPs, though their budget share was much smaller (5;2 %). While their participation was valuable, in some cases they lacked competencies and resources. NMS role was as followers rather than leaders. The importance of third country particiation was not valued as high as general international cooperation, but still participation was largely optimal. The benefits to European projects from third country participants resources and knowledge was high, as well as benefits for the third country partners. The international standard of third country participants was high, according to the respondents to the survey. 57

58 6.5 SME participation The evaluation questions that are answered under this section are focused on the extent to which SMEs participated in the FP6 TP5 projects, via which instruments and what characterized their participation. We first present some statistics. 23,4% of the participants were companies (Business Enterprise Sector - BES, including both small and large companies). Across the types of instruments, the share of companies was highest for Coordination Actions with 31 %, while the NOEs and IPs engaged 24 and 23 % respectively. STREPs had the lowest share of company participants with 11 %. The participation of the BES in Thematic Priority 5 is low when compared to the average BES participation of 35 % computed across all Thematic Priorities. In terms of BES budget share, TP5 ranks second to last. Less than 14% of the TP5 budget went to the BES, which is 21 percentage points below the average BES budget share (35,0%) (computed across all Thematic Priorities). Figure 6 1 and Figure 6 2 show estimates 17 of the SME participation per Thematic Area, in percentage of total participants (per Thematic Area) and in budget share: in both cases, TP5 scores below average. Especially the proportion of the budget that was allocated to SMEs under TP5 is quite low: only 6,1% of the budget went to SMEs under TP5 whereas, on average, across all Thematic Priorities, 10,2% of the budget was assigned to SMEs. Only TP7 (Citizens and Governance) scores lower in this area. Figure 6 1: SME (estimate) participation in FP6 priority areas Source: FP6 Final Review: Subscription, Implementation, Participation, European Commission, Research Directorate-General, Evaluation and Monitoring of Programmes, Brussels, June 2008, p There are no complete and validated data on participants SME status. To overcome this difficulty, the EC produced estimates by (a) defining the population of potential SMEs among FP6 contracts and (b) assigning SME status to individual participants on the basis of information during contract negotiations or self-declarations provided by applicants at proposal submission. (Source: FP6 Final Review: Subscription, Implementation, Participation, European Commission, Research Directorate-General, Evaluation and Monitoring of Programmes, Brussels, June 2008, p.15). 58

59 Figure 6 2: SME (estimate) budget share in FP6 priority areas Source: FP6 Final Review: Subscription, Implementation, Participation, European Commission, Research Directorate-General, Evaluation and Monitoring of Programmes, Brussels, June 2008, p.17. Generally, 40 % of the survey respondents found that not enough SMEs were involved in the project, while 54% found that the optimal number of SMEs were involved. Focusing on SME involvement, then about 70% of the respondent s state that it is rather important or even crucial that SMEs are involved in the project in regards to the achievement of the project objectives. Companies are, on average, more convinced of the importance of SME-involvement (with a mean of 2.25 compared to the overall average of 1.94). More than one third of the higher education institutes consider the involvement of SMEs as rather unimportant. Table 6 5: Importance of SME involvement - by type of partner (a) Higher Education Institutes (n=233) Research Organisations (n=255) Companies (n=90) Other (n=41) Total (n=619) n (3) Crucial 21.5% 18.0% 40.3% 27.3% 23.0% (2) Rather important 44.5% 53.1% 44.2% 42.4% 48.0% (1) Rather unimportant 34.0% 28.9% 15.6% 30.3% 29.0% Total 100% 100% 100% 100% 100% Mean (a) How important was the involvement of SMEs in the project for the achievement of the project objectives? Means with 3 = crucial; 2 = rather important; 1 = rather unimportant. Source: IDEA Consult on the basis of survey results The case results support the survey results. However, results are more diverse on instrument level. For IPs about 8 out of ten find the SME involvement crucial or important, while it is a bit less 7 out of ten for STREPS. More than half of the NOEs find SME participation unimportant or not even relevant for them. Overall, many participants acknowledge the specific contribution from both SMEs and large companies. While the SMEs in particular contribute with specialists, the larger companies often make (food) material, specialist equipment and research infrastructure accessible for other participants. However, when it comes to the 59

60 agricultural aspects of the food quality and safety there are only few large companies that are relevant regarding partnerships, as the sector mainly comprises SMEs. The following box illustrates the characteristics of SME partners in the case projects. Box 10: SME characteristics - examples from the case studies PATHOGENCOMBAT (IP) is a successful project, where a particular feature is that it has many SMEs involved. These SMEs are likely to benefit from increased competitiveness the economic impact can be expected to be high. PATHOGENCOMBAT is a highly relevant project, as expressed by the expert: the combating of food borne pathogens is probably never ending with new and emerging pathogens entering the food chain. The success of PATHOGENCOMBAT includes contributing substantially to improving effectiveness and uniformity in reducing the prevalence of food born pathogens in European food and has created an important foundation for scientifically based food safety management in Europe. The outputs are expected to be widely exploited by SMEs to produce safe food and improve health and well-being of the European citizens, as well this will increase competitiveness of the industry. As pointed out by the expert, almost half of the partners in PATHOGENCOMPABT were SMEs, which is very promising for the usage of the results, including the development of hygienic design and new processing technologies to prevent settling and spreading within the food chain. In other words, PATHOGENCOMBAT has, by including a large number of SMEs increased the likeliness for economic impact of its results, including facilitating proactive, coherent, effective and flexible responses to new/emerging pathogens. For the industry such knowledge will provide a clear competitive advantage. PATHOGENCOMPAT is an example that supports the inclusion of SMEs. In IMAQUANIM (IP), the SMEs' objectives for participation were mainly the opportunity to cooperate with qualified researchers nationally and internationally, and to focus on specific research tasks. In addition, participation in the project was good publicity, and the SMEs were able to broaden their network in the field. The SMEs tended to have R&D capacity. Same was evident for SMEs in SEAFOODPLUS (IP). From ISAFRUIT (IP), the interviewed SME s key competences were nutrition and health benefits from food. Consequently, the company had already conducted studies to document the link between fruit and health, including research into the functional effects of ingredients. In that sense, the company had experience with R&D activities. There are several cases where the SMEs are not directly research related. Three SMEs were partners to ECNIS (NOE). Two SMEs addressed communication and one participated in research activities. The SMEs were key players in achieving project objectives. To illustrate this, one of the SMEs in communication implemented a video conference network covering all ECNIS partners, aiding the integration of the partners by facilitating interaction. The SME involved in research activities was important as it constituted a direct way of converting ECNIS research results into business opportunities. The two SMEs in Ga2Len (NOE) were not directly important to achieve the project objectives, however, they still were key to achieving the objectives, via the provision of IT and dissemination activities. A challenge was that it was difficult to integrate SMEs. The reason for this was that the scope of the project to establish an EU wide allergy and asthma research network was not so suitable for SMEs. It appears that there is a tendency for SMEs participating in IPs having a higher level of research capacity, while this differs in the NOEs, where in several cases SMEs are service providers. The participation of SMEs differed somewhat between especially IPs and NOEs. IPs have in general been more successful in this respect, one reason being that the research targets of IPs are more clear than those of the NOEs, and industry needs clear targets in order to get management approval for the funding that they will use. IPs have generally also taken more initiatives to support the integration of SMEs, e.g. industrial platforms that eased contact to many companies without direct participation, national support points, e.g. within the research organisations, that 60

61 were able to assist the SMEs with the administrative issues of their participation. One project also used national contact points at the local partner institutes to communicate to SMEs, as the SMEs were not able to travel to other countries to participate in seminars and workshops. The following box presents more details on the role SMEs took in the projects and the importance of SMEs to the projects, as well as other challenges to cooperation. Box 11: Participation of SMEs and their role - examples from the case studies There were seven SMEs involved in HEALTHGRAIN (IP). They all had a scientific focus. In the case of the SMEs, cooperation proved difficult. Initial expectations to them were high; however, they were not able to deliver the expected high quality output. The main reason was the SMEs lacked the required competences. An important project objective of IMAQUANIM (IP) was to involve SMEs to ensure that research outputs were converted into practical applications, but cooperation turned out to be challenging. Five SMEs participated in the project from the beginning, but two SMEs exited the project before full project completion. Reasons given were that one SME was very small and in a financial situation that precluded further participation. Another SME found that the market was too small and that the research activities were too far from the market. However, it was not an easy task to optimize the involvement of SMEs. One SME did not produce as many outputs as expected because other project partners did not manage to take advantage of the opportunity to cooperate with an SME. Twenty of the ISAFRUIT (IP) partners are SMEs. Cooperation is working quite well; however, a challenge is that the SMEs are very narrow in their scientific focus and mainly oriented towards their country of origin. Another significant challenge is the administrative process. Often the SMEs lacked the motivation to handle such issues, especially reporting requirements. In order to solve this, the ISAFRUIT management established national contact points at the universities to assist the SMEs. Some SMEs chose not to participate in the project due to the time horizon of the project. SMEs have a shorter time horizon compared to universities, and they do not wish to engage in activities where the results lie too far in the future. The PATHOGENCOMBAT (IP) project had a large number of participating SMEs 17 out of 44 partners. In order to secure proper integration of the SMEs, each SME was assigned to a national research centre. The reason was that the SMEs were not prone to participate in common meetings, instead the information flow to the SMEs took place via the national centres. In several cases, this model was highly successful, and Belgium ended up establishing a network of 90 SMEs, which were ultimately gathered in a new project. In BIOCOP (IP) 9 SMEs were involved. One SME was involved for IT related issues such as the website development. Others acted as providers of specialist reagents and equipment and finally there were also researching SMEs. All interviewees agreed that the involvement of these SMEs is crucial for the reach of the objectives. As most of the SMEs involved are spin-outs from Universities, they are specialists and are extremely good in their field of expertise. One interviewee considers the involvement of SMEs as a buy-in of specialist skills that you cannot get elsewhere. As summarized by one respondent: Overall, the success of SME involvement depends on several factors. One factor is the structure of the SME and its economic potential; another factor is the high commitment of the management. Finally, the topic of the project and the timing must match the objectives of the SME in full. These factors should be considered carefully in the initial screening of SMEs suited for similar projects. In general many suggestions were made to ease SME participation. The suggestions included developing a 'light' version of SME participation that would not have as big reporting requirements. Another idea was to make a database with 'SMEs with FP experience'. The database would make it easier for coordinators to find reliable SMEs and SMEs would be more visible to potential projects. Finally, it was suggested to establish an industry advisory board instead of direct participation. Another difficulty connected with involving SMEs and other companies in a university/research organization dominated collaboration is the fact that researchers want to publish their results while industry wants to pursue patents. In this context there is also a difference in time horizons, researchers having long 61

62 time lines and not being strung up on presenting results tomorrow, while SMEs must develop their businesses constantly and new initiatives must pay off within a shorter time. Some participants feel that due to the SME requirements by the EC they were forced to include SMEs, while larger companies would have been more appropriate partners. Compared to the other thematic priorities, the Business Enterprise Sector and the SMEs in particular, were underrepresented as participants in TP5. Nevertheless, the involvement of SMEs is regarded as important to achieve the project objectives. SME participation was most successful in IPs. SME participation was overall challenging to the projects and several suggestions were put forward to better integrate the SMEs. 6.6 Gender balance The evaluation questions that are answered under this section are focused on what role gender has played in the projects. Figure 6 3 gives an overview of the share of female scientific coordinators and scientists in charge of funded projects, organised per Thematic Priority. When compared to the other Thematic Priorities, it is clear that TP5 attracts a high percentage of female researchers and in leading positions: 1 in 5 coordinators of TP5 projects are women and nearly 30 % of scientists in charge are female researchers. Only TP7 (Citizens and Governance) scores higher in this respect. 62

63 Figure 6 3: Gender Distribution of Scientific Coordinators and Scientists in Charge - by Thematic Area Source: Gender Equality Report-Framework Programme 6, European Commission, Research Directorate-General, Scientific Culture and Gender Issues Unit, October 2008, p.15. From the case studies, it was evident that most projects did not actively approach gender balance issues. Overall, in almost 3/4 of the cases no specific initiatives were taken towards female participation on top of the required monitoring. The IPs stand out as these projects did take initiatives towards female participation in almost half of the cases. Box 12: Gender balance examples from cases ECNIS (NOE) set up a gender issue panel that should provide advice on how to promote female participation, especially for management and leadership positions. Activities of the panel include mentorship programme for female leaders, career development initiatives, and a leadership training programme for female PhDs. However, it was found that the female PhDs did not wish to participate in the leadership training programme, so it was never completed. ISAFRUIT (IP) formulated a gender policy to address the issue of gender balance in the project. Only one woman was on the management team of the project. Efforts were consequently made to recruit women for the positions of external advisors to be included in the management group. Two out of three external advisors were female. Specific attention was paid to the involvement of women in BIOCOP (IP). In the beginning of the project they assigned a gender manager. The first idea was rather to monitor and it was mainly introduced in the project as a strategic choice to try and get funding. However, it appeared to be a very successful and effective strategy during the project. They established a fellowship scheme only open for female researchers within the consortium and it made it possible for them to visit other organisations for a period of 2-4 weeks. The fellowship affected many PhD students and is a boost for career opportunities in the future. In addition to that, they also put women on key positions in the project. SABRE already exceeds the 40%-target of female scientists involved in the projects. Female researchers are active at all levels, from early researchers to WP leaders. However, at the start of the project, the proportion of experienced female researchers was still limited. Therefore the specific Gender Action Plan (GPA) aimed to increase the number of women developing careers in science and taking on roles at higher levels within SABRE. The task of this GPA-manager is to coordinate the networking with the female scientist networks and to check project documentation for gender aspects. 63

64 As in the majority of the projects, no major initiatives were taken to address the gender balance in HEALTHGRAIN (IP). At the initial mapping of the gender balance, female participation was very high, at about 80 %. The coordinator and administrative manager were both female. Female researchers have been very active in the project, e.g. a female Hungarian researcher is the first coauthor on many scientific papers, and therefore the project chose not to engage actively in female participation. The main reason for this low level of initiatives was that the female participation in most cases was regarded as sufficient or even higher comparatively to the male participation. Though, often the gender balance is different across the career levels. At PhD and student level the female participants often dominate, while at the higher management levels, male participation is comparatively higher. About one-quarter of the case projects did actively pursue gender initiatives. Such initiatives and gender action plans include fellowship programmes focused on female participants, encouraging women in management positions, increase of networking among the women, introduce flexible working hours and family friendly policy, developing key indicators to monitor female involvement, encourage female scientists to actively promote science as a career for women in Universities, and mentorship programmes. Overall the available statistics show that TP5 attracts a high percentage of female researchers and in leading positions. It is therefore no surprise that gender issues played a minor role in the case study projects. 64

65 7 PROJECT OUTPUTS AND RESULTS 7.1 Introduction In this chapter, we will give an answer on the following list of evaluation questions: EQ 17: What have been the main objectives to participate? EQ 18: What has been the true motivation for the presence of private entities in the project? EQ 19: Did the projects attain their original aims? EQ 20: Did the projects constitute the best way of obtaining the objectives set (in terms of chosen instrument, research approach,...)? EQ 21: What were the main projects outputs and results that were generated by the projects, and what is the overall quality and relevance (or usefulness ) of these outputs? EQ 22: Were the individual project deliverables delivered on time? EQ 23 Were the projects cost-effective? EQ 24: To which extent have the outputs and results been disseminated (also in terms of geographic scope) and how effective have the project dissemination actions been? EQ 25: Were the individual project deliverables easily accessible to peer groups? The answers on these evaluation questions are discussed under the following sections: Main objectives to participate Reach of the project objectives Project outputs and results Dissemination of the project results Cost-effectiveness 7.2 Main objectives to participate We briefly summarise the main objectives for participating in the FP projects, like indicated by the interviewees in the case studies. There are several reasons for participating in the FP projects: 65

66 - Research/science driven objectives: o Interest in the research content, because it is closely related to the research content of the organisation; o Large scale and scope of research: sharing of expertise from different organisations, sharing of tests, samples, experiments from different countries; o Multidisciplinary character: combination of different science-areas, combination of basic, partially applied and very applied research; - Policy-driven objectives: o o Need for integration and/or harmonization across and even beyond the EU; Highly placed on the political agenda; - Organisation-/personal driven objectives: o Increased visibility; o Enlargement of contacts with other research-oriented or industrial organisations across (and even outside) Europe; o Increase of international collaboration; o Increased access to knowledge, scientific excellence, technologies, methodologies, infrastructure; o o o Involvement in training programme and increase of human mobility; A large amount of money for research; Vehicle for additional funding (in the form of follow-up projects, spin-offs,...) Box 13: Main objectives to participate examples from cases Research/science driven objectives QualityLowInputFood (QLIF): The project addressed the most important research questions on organic farming of the institutes involved. In addition, the project provided access to new research, experience, methods, new partners,... It guaranteed intensive information exchange with most of the partners in the consortium, both during and after the project, such as the sharing of research methods, concepts, etc.. Research is conducted on a larger geographic and budgetary scale, with a broader scientific spectrum, including a large number of experts across and even outside Europe, which is an important difference with projects funded at the national level, which are generally much smaller. And finally, it was possible to compare national results with other countries. Policy-driven objectives Feeding Fats Safety: The main reason for the interviewees to be involved in the project was the topic. There was a lack of standardization and knowledge within the specific field of the project. One of the interviewees felt responsible as public body to take this opportunity of participating in an EU project in this area in order to clarify the market in terms of definitions, technological lines of production and classification. For the IAG involved, the project was important because of its aim to integrate both producers and consumers. Participation in this project helps them to provide relevant information to the members of the organization which is in the end their main goal. Organisation-/personal driven objectives Parasol: The achievement of a high level of research interchange and collaboration, increased network activities, the possibility to combine knowledge and expertise of different institutes on a larger global scale and improvement of the organisation s profile were the most important motivations to participate. Isafruit: For the interviewed company, the rationale for participating was to create links to customers, i.e. the large food production companies, and to enhance credibility and market and sourcing opportunities. 66

67 7.3 Reach of the project objectives Reach of project objectives At the project level First, we present the main conclusions on the realisation of the project objectives at the level of the project: Figure 7-1: Reach of the project objectives at the project level (n=728) Completely 24.0% To a large extent 61.3% To a limited extent 9.5% Not at all Too early to assess 0.1% 5.1% 0% 10% 20% 30% 40% 50% 60% 70% Source: IDEA Consult on the basis of survey results Overall, the survey-respondents indicate the project objectives have been (or will be) reached to a large extent (on average). About one out of four respondents even indicated project objectives have been reached completely. Only one out of ten respondents indicated they have not at all been reached or only to limited extent. There is a slightly higher reach of project objectives in the 30 cases. The status of the project (finished or ongoing) has no (or only a very limited) impact on the (expected) reach of the project objectives. Not surprisingly, coordinators respond closer towards a complete reach of the project objectives (with 36.7% of the coordinators indicating the project reached its objectives completely). Regular partners, on the other hand, have the largest share (15%) of respondents indicating project objectives have been reached only to a limited extent. By type of organisation, research organisations and higher education institutes are most positive about the overall reach of the project objectives, while companies are least positive. Looking at the different instruments, there are only small differences. Further we present the most important comments provided by the case study interviewees: In several projects, the research objectives were succeeded, but an extension of the project for further experiments, valorisation, monitoring, follow-up activities or dissemination would be very supportive. Several interviewees mentioned that it is regrettable that some very successful projects, that have reached all (or most) of their objectives, must stop at the end of the FPfunding period, because no additional funding can be found. 67

68 Some of the objectives are quite intangible, or it is at least difficult to measure whether they have been achieved (e.g how to measure increased integration, and more problematic, how to attribute this increased integration to the specific project). Several projects required the removal or adaption of project objectives, or the team just failed to reach the project objectives, because of several reasons: o o o o o The rigidity of FP-programme: A proposal is written 1 to 3 years before the start of the project. In addition, the projects have a duration of 3 to 5 years. It is very difficult to predict all project stages for a future long term research project in advance. The reasons for these difficult predictions are unexpected or unforeseeable technology changes, food scandals, changes in prices of material, or the lack of access to the required data, samplings and resources. Cuts in the budget during the negotiation stage or during the first stage of project; Unexpected costs, for example experiments that turned out to be too expensive; Transfer to industry and implementation of the results by industry takes time. Project objectives related to practical implementation and application are often not entirely achieved, because within a 3 to 5 year project, it is very hard to convince the industrial community from the benefits of the results. A comparable problem is the huge time lag between the finishing of a research paper and its publication in a high level journal, which can take several years. Differences in interests. For any (longer-term) research project, the following statement holds true: if you are able to reach 100% of the pre-set deliverables, you did not try hard enough to do something that was difficult There were also some instrument-related problems hampering the reach of the project objectives: o Problematic in some IPs was the integration of interdisciplinary work packages where the consortium was too big or where the linkages between the different work packages were too artificial or even nonexisting; Example - QualityLowInputFood: They planned to work interdisciplinary over the whole period of the project. But there were so many partners, that in the first two years of the project, there was only focus on the disciplines on itself. The focus on interaction (between natural sciences, socio economic analysis, quality management,...) only started in the second part of the project. It would have been better if this had taken place earlier. o Problematic in some NoEs was the fact that too much emphasis was put on the realisation of objectives that were related to interaction, collaboration and network increase and that too little emphasis was put on the other objectives that were more research and industry-oriented. 68

69 Example - Under EURRECA, they wanted to do a very profound analysis on different nutrients, but because of budgetary restrictions this was only possible for a limited number of nutrients. In addition, there was not enough money available to investigate important research gaps. On the other hand, in several projects more objectives were reached than originally planned. Example - Sabre: The money that came available because of falling technology costs was reinvested. At the level of the organisation Next, we give an overview of the main results on whether the individual research organisations have reached their personal/organisational objectives in the project. Figure 7-2: Reach of the organisation s project objectives (n=732) Completely 37.8% To a large extent 50.8% To a limited extent 7.9% Not at all Too early to assess 0.8% 2.6% 0% 10% 20% 30% 40% 50% 60% Source: IDEA Consult on the basis of survey results With an overall mean value 18 of 3.29 the survey-respondents agree that their organisations have reached their project objectives to a large extent. Note that the respondents are more positive about the reach of the project objectives at the level of their organisation than at the overall project level (means 3.29 and 3.12 respectively). This positive result is confirmed by the case studies (with 94% of the interviewees indicating that their objectives are reached to a medium or high extent). According to the case study interviewees, the main difference compared to the objectives at the project level is the fact that the organisation s objectives put more emphasis on the organisation itself and less on research. Examples of typical organisational objectives to participate are an increased visibility, an increase of the knowledge base, the enlargement of the organisation s network, increased collaboration and integration into the scientific community. In terms of instruments there are only small differences. Not surprisingly, the coordinators indicate their organisations have reached their project objectives to a large extent, with a tendency towards a complete 18 Means with 1 = not at all; 2 = to a limited extent; 3 = to a large extent; 4 = completely. 69

70 reach of their organisation s objectives (mean value of 3.45). Regular partners show the lowest mean value of (still) It seems that the more involved (in coordinating tasks) in the project, the more survey-respondents are satisfied with the reach of their organisation s project objectives. This finding is confirmed by the case study results. By type of organisation, the higher education institutes and the research organisations show a stronger tendency towards a complete reach of their organisation s project objectives (mean values of 3.31 and 3.35 respectively) opposite to the companies and the rest category others (mean values of 3.17 and 3.09 respectively). Despite the low sample sizes of some of the categories, these results are confirmed by the case studies. In conclusion, the project objectives both at the level of the project and at the level of the organisation are reached to a high extent. But did the projects constitute the best way of obtaining the objectives set? Suitability of the project Important project features We presented the survey-respondents 19 a list of (potential) project features and asked them which of these features contributed to the achievement of the project objectives. As such, the responses to this question allow saying something about the project features contributing to the successfulness of projects. Table 7 1: Effectiveness project features for objectives achievement (a) Total n Mean % no opinion Pooling of complementary knowledge % Pooling of qualified personnel % Pooling of complementary data % Pooling of complementary methods and techniques % Pooling of financial resources so as to achieve critical mass % Pooling of advanced equipment/infrastructure % Sharing of risks % Pooling of IPRs % (a) Which of the following project features contributed to the achievement of the project objectives? Please indicate to what extent you agree/disagree. If a specific project feature does not characterize your project, please tick not relevant. If a specific project feature does characterize your project, but you don t have an explicit opinion on it, please tick no opinion. Means with 1= completely disagree; 2= disagree; 3= neutral; 4= agree; 5= completely agree. Source: IDEA Consult on the basis of survey results Overall, the respondents agree that 6 out of the 8 project features contributed to the achievement of the project objectives (with mean values around 4 or more). The project features that overall contribute most to the success of 19 We only asked this question to the respondents who indicated in a previous question that their project reached its project objectives at least to a large extent, which explaines the lower number of respondents. 70

71 projects are the pooling of complementary knowledge 20 and the pooling of qualified personnel. Generally speaking, all project features identified as important are in accordance with the traditional European Added Value arguments justifying intervention in the field of research at EU level. This is also true for each of the different instruments with the exception of STREPS where the pooling of complementary methods and techniques is the second most important project feature contributing to the success of the projects. Of the indicated project features, the pooling of IPRs and the sharing of risks contributed the least, overall, to the achievement of successful projects. But these are also the project features with the highest number of respondents with no opinion (and in fact also with the highest number of respondents with not relevant because the project feature did not characterise their project). Box 14: Importance of pooling of knowledge: expert assessment of the cases - example EU FRESHBAKE (STREP) is an example of a project, where the relevance was high (bake off technology and nutritional aspects), but the outcomes of the project only medium successful due to the lack of competences among the project participants. In regards to the relevance of the project, the expert noted that part of the argument for the project relating to the nutritional aspects of bake off was not really justified in the competences of the project participants. The technological part of the project was good; however, the expert questions the quality of the results related to the nutrition part, hereunder e.g. the glycaemic index and the usage of this, as its valid usage is still under strong scientific debate. Overall, the expert questions if the methods are used correctly and therefore whether the results are valid. In other words, the nutritional aspects were below expected levels. The reason for this could be lack of competences among the nutritional project participants. Overall, the expert finds that the project should still have been approved, but with focus on the technological aspects solely. The project still contributes to the knowledge base in the EU from its publications, and it contributes to a number of strategic and specific objectives. Suitability of the instrument In the case studies, we asked the interviewees whether the type of instrument (NOE, IP or STREP) under which the project was implemented was suitable given the research needs / topic of the project. Each case study responsible made an overall assessment based on the answers of the four interviewees. In 90% of the cases the chosen type of instrument was considered to be the most suitable type. This share was slightly higher for IPs (93.3%) compared to NoEs (87.5%) and STREPs (85.7%). An important advantage of an IP is that it allows to integrate across different research disciplines and across different types of research activities (from basic science to applied research and in the end to technology transfer). In addition, IP-instruments seem to be very advantageous in terms of different types of economies of scale: 20 Box 14 gives an example of a case study project where the lack of competences and complementary knowledge led to results that were below expectations. 71

72 one coordinator instead of several coordinators of sub projects; Some underlying technologies are the same (synergies); Relevant sizes of experiments and collaboration on these experiments across different countries; Appropriate number of partners and higher chances to get involved with unknown organisations/people; Volume discounts (up to 50 %) because orders of tests can be combined and consumables and test material can be purchased in bulk; Possibility to bring cutting edge research together and to have top speakers on workshops and conferences. The large size of the budget allows that even the partners with only a small bit of the cake can contribute to and benefit from the project in a substantial and useful way. One is also more likely to collaborate with partners with complementary knowledge. A disadvantage of an IP is the involvement of too many different agendas, e.g. the inclusion of sufficient NMS and third country participants. This does not always work well. Because of these agendas, some of the partners are included for the wrong reasons: not because of their value added, but just to win the bid. In addition, the research dealt with under an IP, can also be too broad, resulting in artificial or even non-existing linkages between the different work packages. Another disadvantage, as a consequence of the mix in types of partners, is that the companies or private sector organisations in the project team have different funding schemes (full cost with co-financing depending on the size) compared to the public sector participants (additional cost financing). Box 15: Suitability of the instrument examples from cases Healtgrain (IP): A NoE would not have been sufficient to handle the proposed work programme. To the industrial sector, it was also an advantage that the project was an IP, which secured concrete project objectives. The project design facilitated industry's possibilities of integrating knowledge and results into business activities. Traceback (IP): The project was able to horizontally integrate partners from different disciplines. Furthermore, the magnitude of the project and the integration between industry and academics support the typification as an IP. Ecnis (NoE): Most interviewees found that the NOE was good for the purpose of gathering a fragmented research area. One of the interviewees mentioned the challenge of maintaining the current level of integration after the five project years. He found that an IP would be suitable for the continuation of the project. In addition, for the SMEs the IP would offer a more attractive scheme as more targeted research goals can be set. The SME is not interested in more integrating activities such as training and exchanges. One partner, however, found that the project ought to have been an IP from the beginning as it would have enabled a research-oriented approach. GA2LEN (NoE): The purpose of the project was to integrate the European research within the field of allergy and asthma. To this end, GA2LEN has achieved the goal and became a sustainable structure, therefore being an NoE must also be said to be the correct funding instrument. Feeding Fats Safety (STREP): A Strep is a good instrument for projects that can be done with a more moderate number of partners which makes it easier to coordinate and manage and also in terms of research topic. The project was quite focussed and not as broad as for some other instruments is required. Furthermore, a Strep was the ideal instrument for the multidisciplinary research project FEEDING FATS SAFTEY was. 72

73 Helena (STREP): Since HELENA was a STREP with the characteristics of an IP in terms of size of the project, the amount of funding received from the EC was not sufficient. According to the interviewees, the problem lied within the call for the project that was too broad and demanded too much for the given type of project and budget. Quality of the research approach In the case studies, we further asked the interviewees how they evaluated the quality of the research approach that was used in order to achieve the project objectives. In most of the projects, this quality was considered to be high (40%) to very high (45%). There is only a small difference between the finished and ongoing projects. By instrument, the quality of the research approach in NoEs is slightly lower than the quality of the research approach in the other two instruments. This is, however, not a surprising result, since NoEs are typically more oriented towards network building and collaboration and less towards scientific and technological outputs, where the quality of the underlying scientific and technological methodologies is more important. By type of organisation, the quality of the research approach is considered to be higher by the interviewees from research organisations and higher education institutes compared to the interviewees from companies. The high quality of the research approach, according to the interviewees, can be explained by the following aspects: In most of the projects, the partners that were in charge of the most important project tasks and that had the most important responsibilities were chosen for their specific expertise, which had a positive influence on the quality of the chosen research approach. The flexibility allowed projects to change the methods and techniques during the project execution. Example - Sabre: The preliminary results in one of the areas showed that what they would have liked to do was not appropriate. It was ethically unacceptable, because the number of animals was far too low to get robust results. As a consequence, they removed the complete experiment from the project. This quick reaction also pleased the external evaluators and the EC scientific advisor. This was, however, not always feasible in all projects. And in some projects, difficulties related to the chosen research approach were only discovered in a later stage or after the project. Example - QualityLowInputFood: There were some problems with the transportation of research materials. Crops of different vegetables were transported from the UK to other countries. The crops were transported in an area that was far too humid, causing huge mould-problems. It was a costly investment, and only a limited share of the crops was usable for further testing. 73

74 A third aspect with a positive influence on the research approach was the provision of trainings and courses of the applied techniques to the other partners during the project to spread the knowledge and to ensure the harmonized character of the methods used within the project. In conclusion, the project objectives both at the level of the project and at the level of the organisation are (expected to be) reached to a high extent, according to the survey respondents. Some of the projects among the case studies succeeded their project objectives. When projects (partially) fail to reach their objectives, the reasons mentioned were the rigidity of the FP-programme, cuts in the budget, unexpected costs, the transfer to industry and implementation of the results by industry taking time and differences in interests. By type of organisation, the higher education institutes and the research organisations show a stronger tendency towards a complete reach of their organisation s project objectives opposite to the companies. The project features that overall contribute most to the success of projects are the pooling of complementary knowledge and the pooling of qualified personnel. Generally speaking, all project features identified as important are in accordance with the traditional European Added Value arguments justifying intervention in the field of research at EU level. Of the indicated project features, the pooling of IPRs and the sharing of risks contributed the least, overall, to the achievement of successful projects. In 90% of the cases the chosen type of instrument was considered to be the most suitable type. In most of the projects, also the quality of the research approach that was used in order to achieve the project objectives was considered to be high (40%) to very high (45%) with the interviewees from research organisations and higher education institutes being most positive. 74

75 7.4 Project outputs and results Main outputs and results Scientific papers or publications in refereed journals and books are most often considered by the case study interviewees as most important output. Other important outputs are new, improved or integrated tools, techniques, models and information systems for data-sharing and new or improved codes, standards and protocols. Outputs considered to be less important are sustained working or expert groups and the establishment or enlargement of a research network. Also the more exploitable and commerciable outputs, such as demonstrators and prototypes, patents and new products or services are less mentioned. We therefore conclude that outputs are mainly research-focused and less industryoriented. We present in the following box a number of concrete outputs of the 30 cases, structured by type of output. Box 16: Main outputs: illustrations from the case studies 21 Scientific papers or publications in refereed journals/books: PARASOL: An edition of Veterinary Parasitology fully dedicated to the results of the Parasol Project ACE-ART: Special edition of the International Journal of Probiotics & Prebiotics (2008, Vol.3, Nr.4) was devoted to summarize the results of the ACE-ART project. New, improved or integrated tools, techniques, models: SABRE: o Exploitable results in numerical genomics: software for QTL analysis, MIXBLUP software and an R package for eqtl analysis o Exploitable results in genomics and bioinformatics: Pig Genome Sequence (Chromosomes 7 & 14) o Exploitable results in the area of product safety: QTL for eggshell quality on chromosomes 2, 3, 6 and 14, SNPs in known candidate genes affecting egg shell quality, differentially expressed genes from shell gland o Exploitable results in the area of animal wellbeing: Genes differentially expressed in Meishan and Large White pig adrenals in basal condition and in response to ACTH o Exploitable results in the area of product quality: 7.5K Pig SNP Panel HEALTHGRAIN: o In the field of plant breeding and biotechnology: a plant breeding toolkit and a 'diversity screen' of amounts and compositions of bioactive components in almost 200 different grain lines (including wheat, rye and other cereals), which serves as input to the development of cereal grain varieties with a higher content of bioactive components. This knowledge will enable farmers and grain processors to buy and grow varieties with higher levels of fibre. Demonstrators, prototypes: Ga2Len: establishment of a novel standard series of European skin prick tests for inhalent allergies. BIOCOP: Prototypes which include at least 2 scientific instruments and at least 4 kits that are ways to deliver rapid analysis on a range of chemical contaminants. New or improved production processes: Eurofir: Design and implementation of a process for the identification, prioritisation, collection and analysis of traditional and ethnic foods. New or improved codes, standards, protocol: 21 A more extensive list of project outputs can be found in the background report. 75

76 FRESHBAKE: proposition of the standardisation of the energy efficiency of baking ovens, freezing chambers, etc.. through a numerical index to assess the energy efficiency of the process EUROPREVALL: the study protocols are probably set to become international standards. Patents or other forms of IPR: Truefood: there were two patents applied for in the project: one on the use of natural disinfectants for milking, another in the field of control of the ripening rooms of traditional cheeses. The patents are owned by a public body for public health and by this research institute, but they license it freely to companies. But in general the outputs are more in know how than in patenting. OPTIMOILS: patent on membrane processe New, improved or integrated information for data-sharing: TRACE: development of a state of the art database on specifications and modelling systems (the TRACE database on molecular markers or food map) so that determining geographical origin will be easy (for about 5 products). HI-WATE: development of a database on prospective cohort with personal exposure evaluation. The database links environmental variables and personal health data for exposure-response assessment (cancer studies need longitudinal exposure data in order to compute accurate disease prevalence and uses retrospective case-control designdrinking water disinfection by products). Set of guidelines or recommendations of best practices: Nugo: Standard Operating Procedures (SOP) and bioethical guidelines for nutrigenomics research in humans. ISAFRUIT: analysis of the health effects of whole fruits and fibres of fruits (residuals from juice production), e.g. demonstrating a direct positive effect on cholesterol diseases. This new knowledge can influence the nutrition guidance not to include juice as part of the recommended 600 g of daily fruit and vegetables. New, improved or integrated facilities/infrastructure: EADGENE has established a virtual laboratory and developed a series of common research tools and platforms, including DNA micro arrays within chicken, bovine and pig, microarray annotation files, bioinformatics tools, and guidelines for addressing IPR issues. ECNIS: development of a video conferencing system, including all ECNIS partners. Sustained working or expert groups: EURRECA: development of a sustainability group to continue the follow up after the end of the project. In the sustainability plan, they are discussing how EURRECA can evolve to a EURRECA 2 (European Institute for Food Policy and Public Health) in order to strengthen the position of the network. Further, collaboration with other networks such as NUGO, EUROFIR and MONIQA is explored, that could lead to a European Academy for Food, Nutrients and Health. Truefood: a spinoff from TRUEFOOD was the establishment of the ETP, Food for Life; and in particular the national platforms on this topic, with now 35 national platforms. Establishment or enlargement of a research network: TRACEBACK: Setup of six Regional Technology Platforms to receive input, provide training, and facilitate dissemination of project outcomes. ENDURE: The main objective being the continuity of the network after the NoE ends, it was decided to continue under the form of a permanent structure under the form of a European Research Group (ERG) for developing tools, competencies, advice to policy (implementation of the EC Directive on sustainable pesticide use), setting up projects, and the virtual laboratory. The coordinator of ENDURE will coordinate this structure with an engineer employed full-time. All partners are contributing in terms of man months. 76

77 For most case study projects, we obtained more detailed data on (a) the number of published or accepted articles that appeared in peer reviewed journals and (b) the number of patent applications, granted patents and exploited patents 22 within each project. Using this case study data, we are able to check (a) whether the instrument under which a project is financed affects the number of patents and publications that are produced under that project and (b) whether finished projects have already generated more of these outputs than projects that are still ongoing. Based on the data from our 30 case studies, we find that, on average, IPs produce the most peer-reviewed publications, followed closely by NoEs: IP projects generate on average 129 peer-reviewed articles; NoE projects deliver an average of 120 publications. The STREPs in our case studies generated much less publications: on average, 25 articles were published per STREP project. Also, projects that were already finished, had more publications (mean value of 109) than ongoing projects (mean value of 92) (see Table 7 2 and Table 7 3). We also checked whether larger projects (measured by the number of partners and the total cost of the project) result in more publications: the data indicate that these two factors are indeed positively related. Due to the variability in the data, we cannot extrapolate these results further for all TP5 projects. Table 7 2: Average number of publications according to instrument Instrument Total IP NOE STREP Average number of publications Source: IDEA Consult on the basis of interviews Table 7 3: Average number of publications according to the status of the project Status Total Finished Not Finished Average number of publications Source: IDEA Consult on the basis of interviews The case study information regarding the number of patents reveal that IPs on average produced the most patents. However, the number of patents is quite small: within IP projects, on average, there had only been 2 patent applications. The NoEs that were analysed in our case studies did not generate any patents. However, due to the large variability in these data, we cannot draw hard conclusions. Finally, the experts that assessed the cases indicated that in their opinion, IPRs should not be developed with public money. This viewpoint aligns with what many interviewees stated in the question of whether results should be published or patented many interviewees chose to publish more Quality of the outputs 22 Examples of exploited patents are: the in-house development / application of a technology (in products, processes, services, ) by one or more of the project partners; the licensing out of a patent; the transfer of ownership; collaboration with a third party 77

78 We first describe the overall results. Figure 7-3: Overall quality of scientific and technological outputs (n = 741) (a) Very high quality 21,7% High quality 60,1% Neutral 14,7% Low quality Very low quality 2,9% 0,6% 0% 20% 40% 60% 80% 100% (a) Overall, how do you evaluate the quality of the project s scientific and technological outputs (scientific papers and publications, patents and IPRs, demonstrators, prototypes, new tools/techniques/models, etc.)? If the project is still ongoing, please evaluate the outputs that have already been realised, or of which you are sure that they will be realised in the near future. Source: IDEA Consult on the basis of survey results With an overall mean 23 of 4, the quality of scientific and technological outputs is considered to be high. By type of organisation, we see only small differences. Looking at the results by instrument, we see larger differences. Respondents participating in IPs and NOEs estimate the quality of their project outputs higher than the overall average (means of respectively 4.12 and 4.15). The differences between the thematic areas are limited. Next, we present the survey results on output quality for each specific type of project outputs separately. By instrument, we report the following for IPs: With mean values of 3.43 and 3.58 patents or other forms of IPR and new, improved or integrated facilities/infrastructure have the lowest perceived quality. This is quite unexpected given the purpose of IPs to support objective-driven research, where the primary deliverable is to generate the knowledge required to implement the thematic priorities. The highest score on quality is given to scientific papers or publications in refereed journals/books (mean value of 4.18), which is perhaps a bit surprising given the expected applied, industry-oriented nature of IPs. On the other hand, and more in line with the rationale for IPs, new, improved or integrated tools, techniques and models (mean of 4.09) are perceived as high qualitative outputs. Looking at the results of the respondents participating in NoEs, we report the following: 23 Means with 1 = very low quality; 2 = low quality; 3 = neutral; 4 = high quality; 5 = very high quality. 78

79 In general, the survey-respondents participating in NoEs give higher scores to the quality of the different outputs compared to the respondents participating in IPs. On top are the establishment or enlargement of research network (mean value of 4.23) and sustained working or expert groups (mean value of 4.22). This is in accordance with the networking and integration objectives of NoE-projects. They are aimed at strengthening scientific and technological excellence on a particular research topic by integrating at European level the critical mass of resources and expertise needed to provide European leadership and to be a world force in a given domain. This expertise had to be networked around a joint programme of activities aimed primarily at creating a progressive and durable integration of research capacities of network partners while at the same time advancing knowledge on the topic. Patents or other forms of IPR have been given the lowest score on quality (mean value of 3.31). But since this is of minor importance for NoEs, this lower position is not a surprise. Looking at the results of the respondents participating in STREPs, we report the following: The scores on quality given by the respondents participating in STREPs are, on average, lower in comparison with the scores on quality given by the participants in the IPs and NoEs. Especially the establishment or enlargement of a research network (mean of 4.06), the set of guidelines or recommendation of best practices (mean of 4.06) and the scientific papers or publications in refereed journals/books (mean of 4.03) are considered to be of high quality. Again, as for the IPs and NoEs, patents or other forms of IPR are considered to be of lowest quality (with a mean of 3.45). Also for STREPs, this is a rather unexpected result. Finally, for the CAs, conferences and seminars are perceived to have the highest quality of all outputs (mean value of 3.92). The expert assessment of the cases, in terms of usefulness and quality of the actual research carried out during the project s lifetime, was high to very high. In a few cases, the usefulness/value of the research was assessed to be a bit lower than the quality of the research. This means that for instance dissemination activities were too low or no sufficient number of articles was published compared to the size of the project, which limits the value that is extracted from the results. Another reason for lower usefulness could be that the project activities were not carried out in the extent envisaged in the application or that the activities had been delayed to an extent that affected the outcome and results. However, in most cases the expert assessors recognise that the research carried out is of high value and quality and as such strengthens the European research area. 79

80 Box 17: Quality of the research: expert assessment of the cases - example BIOCOP (IP) was assessed to be a successful project. The reasons for this positive evaluation were that the project was relevant it addressed some of the most important contaminants of concern at the time of the application. Also the usefulness of the project was high the output according to number of publications was good (not excellent), and it was concluded that the project has contributed to a significant advancement in the research field. Some delays were noted; however this was deemed to be normal for this type of projects, as scientific research and development is not free of risk. The quality of the scientific results was assessed to be of high standard, e.g. because of publications in high profiled journals. Another important observation was that many of the papers produced were an obvious result of interdisciplinary collaboration with the consortium that has contributed to collaborative links of added scientific value. The expert also expects an impact on legislation, because the project developed better and more affordable methods. Also the project can contribute to increased awareness of the contaminants in the food chain, impact control methods and increase the health of food. Concerning the economic potential, then several methods offered potential for commercial development, which would fit the market needs for affordable diagnostic tests or methods of analysis. This would also allow for more frequent testing, in the interest of all. It was noted that there is still a way to go for this potential economic impact to materialise. Overall, the project contributed to most strategic and specific objectives. A study by the European Policy Evaluation Consortium evaluated the quality of the researchers that participated in projects under the different Thematic Priorities. They assessed the impact of the scientific output of the various FP6 participants by investigating the publication patterns of the lead scientists, i.e. those scientists in FP6 projects that are more likely to publish in the best journals. Figure 7 4 gives an overview of the relative citation performance of FP 6 lead scientists, classified according to the Thematic Priorities 24, Projects of priority 6 Sustainable development, global change and ecosystems have been divided into 4 sub-priorities: Global change, Transport, Energy and Others (including Hydrogen and TREN calls). (EPEC, 2009). For the definition of this indicator, see background report. 80

81 Figure 7 4 shows that the relative citation rate of the Food Quality and Safety Priority scores higher than one, meaning that TP5 s lead scientists publications have a greater citation performance than all articles published in the same group of journals. TP5 ranks lower than most other Thematic Priorities. However, the differences between TP5 and the other Thematic Priorities that are ranked higher are rather small. Figure 7 4: Lead scientists 2-year relative citation rate (per thousand) for all scientific fields taken together, by priority and year of publication ( ) Source: Bibliometric profiling of Framework Programme participants, European Policy Evaluation Consortium (EPEC), Brussels, 19 June 2009, p.78. Scientific papers or publications in refereed journals and books are the most important output. Other important outputs are new, improved or integrated tools, techniques, models and information systems for data-sharing and new or improved codes, standards and protocols. The more exploitable and commerciable outputs, such as demonstrators and prototypes, patents and new products or services are less mentioned. We therefore conclude that outputs are mainly research-focused and less industry-oriented. The expert assessment of the cases, in terms of usefulness and quality of the actual research carried out during the project s lifetime, was high to very high. Finally, the relative citation rate of the Food Quality and Safety Priority scores higher than one, meaning that TP5 lead scientists publications have a greater citation performance than all articles published in the same group of journals. But TP5 ranks lower than most other Thematic Priorities. 7.5 Dissemination of project results We discuss to what extent the outputs and results have been disseminated, also in terms of geographic scope, and how effective the project dissemination actions 81

82 have been, starting with the survey results concerning the extent of dissemination (see table below). Table 7 4: Extent of dissemination of research results and project outputs (a) Total (736) N 713 (1) Not at all 0.4% (2) To a low extent 5.9% (3) To a medium extent 40.4% (4) To a high extent 53.3% Total 100% Mean 3.47 (a) To what extent are/were the research results and project outputs disseminated? If the project is still ongoing, please consider dissemination of the results and outputs that have already been realised or of which you are sure that they will be realised in the near future. Means with 1 = not at all; 2 = to a low extent; 3 = to a medium extent; 4 = to a high extent. Source: IDEA Consult on the basis of survey results In general, we see a positive picture concerning the dissemination of research results and project outputs (overall average of 3.47 with 53.3 % of the respondents indicating results and outputs were disseminated to a large extent). The mean value of the coordinators is slightly higher than the mean of the other partners. It seems that the less involved partner categories are more critical about the extent of dissemination. By type of organisation, we see small differences except for the companies among the respondents. The latter category has a relatively larger share of to a medium extent and a relatively smaller share of to a high extent opposite to the other types of organisations. By instrument, we see that with a mean value of 3.27, the dissemination happened rather to a medium extent for respondents participating in CAs. Respondents from NOEs (mean value of 3.58) and IPs (mean value of 3.55) consider the dissemination to have happened to a medium extent with a tendency towards to a high extent. Looking at the results by thematic area, we see that there are no big differences. The results from the case studies confirm the high extent of dissemination. For the NoEs and especially for the STREPs, the extent of dissemination is relatively lower in the cases, while the opposite holds true for the IPs. Thus, from the survey and case studies results, we can conclude that dissemination happened/is happening from a medium to high extent. But was the dissemination strategy also effective? From the online survey, again, a very positive picture arises with overall about 91% of the respondents claiming that the dissemination strategy has indeed been effective. Coordinators are even more convinced with a share of 95%. By type of organisation, there are only small differences between the higher education institutes, research organisations and companies. 82

83 The case studies confirm the positive results of the online survey. There are relatively less STREPs with an effective dissemination strategy compared to the other instruments. In addition, in 85% of the cases the individual project deliverables were accessible to peer groups, for instance through publications on websites. The following list presents the main dissemination activities that were mentioned during the case study interviews: All 30 case study projects have a website, which is still active. For only two of them the website is not up to date. National and international conferences (towards the research community, the press, consumers, industrial associations,...); Presentations (oral and poster presentations); Scientific meetings; Stakeholder consultations; Technology transfer platforms; National and international workshops (also outside the EU) and seminars; Training sessions and lectures; (e-)newsletters; Articles for consumer magazines; Published papers, articles, booklets, project reports; Leaflets, video, DVD,... We also analysed the geographic scope of the dissemination activities 26. Figure 7-5 presents the results of the online survey. Figure 7-5 Geographic scope of dissemination (n = 710) (a) EU15 EU10 68,7% 81,4% Third country Candidate country 47,0% 43,4% 0% 20% 40% 60% 80% 100% (a) What was/is the geographic scope of the dissemination activities? (multiple answers are possible) If the project is still ongoing, please evaluate the dissemination of the results and outputs that have already been realised, or of which you are sure that they will be realised in the near future. Source: IDEA Consult on the basis of survey results In general, EU15 countries, followed by EU10 are reached the most by the dissemination activities as indicated by respectively 81% and 69% of the respondents. Third countries fell within the scope of the dissemination 26 The survey-respondents that indicated the results and outputs of the project were not disseminated at all (a limited number) did not have to answer this question. 83

84 activities according to 47% of the respondents, a percentage which is, surprisingly, higher than for the Candidate Countries (ticked by 43% of the respondents). Differences in opinion arise when looking at the results by role of the respondents in the projects. Coordinators systematically estimate the geographic scope of dissemination wider compared to the regular partners or the partners with a coordinating role. We further analysed to what extent the geographic scope of the dissemination activities depends on the involvement of countries in the project from a specific country group. It is clear that in case a country from a specific country group is involved in the project, that country group is being reached more in the dissemination activities. In relative terms, third countries are the least reached by dissemination activities even when a third country is involved in the project. The case studies results are much in line with the survey results. In the case study interviews, we specifically asked the interviewees to what extent the Old Member States (EU15), the New Member States (EU10), the Candidate Countries and the Third Countries (including the Associated Countries) were reached. The results are presented in the following table. Table 7 5: Geographic scope of dissemination OMS (n = 30) NMS (n = 28) CC (n = 26) TC (n = 29) Not at all To a low extent To a medium extent To a high extent Total Source: IDEA Consult on the basis of case studies The OMS are reached the most, which confirms the results of the online survey: in all cases the OMS were reached by the dissemination activities (76.7% to a high extent and 23.3% to a medium extent). The results for the other regions are lower, with 29% of the NMS, 34.6% of the Candidate countries and 20.7% of the Third countries only reached to a low extent. In addition, 11.5% of the cases did not reach the Candidate countries at all, and 10.3% of them did not reach any Third Country. These results are not remarkable, since dissemination activities mainly took place in the partner-countries. In addition, dissemination activities such as workshops and conferences are more expensive in countries that are situated at a longer distance, due to increased transport costs and travel time. Overall, we conclude that dissemination happened/is happening from a medium to high extent. This dissemination is also effective as confirmed by 91% of the respondents in the survey and by most of the case study interviewees. In terms of geographic scope, EU15 countries, followed by EU10 are reached the most by the dissemination activities as indicated by respectively 81% and 69% of the respondents. Third countries fell within the scope of the dissemination 84

85 activities according to 47% of the respondents, a percentage which is, surprisingly, higher than for the Candidate Countries (ticked by 43% of the respondents). The case studies results are much in line with the survey results. 7.6 Cost-effectiveness There are some methodological problems related to the assessment of the costbenefit ratio. First of all, several projects were not finished yet or only finished recently at the time of the interviews. Therefore, it is too early to assess all costs and benefits. Second, some costs and benefits are very difficult or even impossible to measure, for example because of their intangible nature. The survey-respondents that participated in finished projects, were asked how they compare the costs and benefits of participation for their organisation (see Table 7 6). Table 7 6: Cost-benefit ratio (a) Total (n=346) n 340 (1) Benefits much lower than costs 2,4% (2) Benefits lower than costs 6,2% (3) Benefits about equivalent to costs 29,4% (4) Benefits higher than costs 44,4% (5) Benefits much higher than costs 15,0% Too early to assess 2,6% Mean 3,63 (a) Overall, how do you compare the costs and benefits of participation for your organisation in this project? Means with 1= benefits much lower than costs; 2 = benefits lower than costs; 3 = benefits about equivalent to costs; 4 = benefits higher than costs; 5 = benefits much higher than costs. Source: IDEA Consult on the basis of survey results On average, the benefits were considered higher than the costs (mean value of 3.63). 59.4% of the respondents indicated the benefits were (much) higher than the costs, while only 8.6% indicated the costs were (much) higher than the benefits. Looking at the role variable, coordinators tend to agree the most that the benefits are higher than the costs. By instrument, respondents from STREPS are most positive about the balance between the benefits and costs of project participation with a mean value of Looking at the cost-benefit ratio by type of organisation, we see only small differences with the higher education institutes leaning more towards a positive cost-benefit ratio (mean value of 3.75) opposite to the companies that are most critical about their cost-benefit ratio (mean value of 3.5). Finally, it turns out that cost-benefit ratios are not much different between thematic areas. 85

86 The following comments from the interviewees in the cases give some indications on how the interviewees perceived the cost-effectiveness of the project: The administrative requirements are perceived to be very demanding by the majority of the interviewees, strongly hampering the cost-effectiveness of the project. Less paperwork and better administrative support could increase the cost-effectiveness. The cost-ineffectiveness of administrative requirements was especially problematic for those partners who were only involved for a minor task in the project. Due to the small amount of budget allocated to these partners, a proportionately bigger part of costs were taken up by administrative work (mainly fixed cost). Consequently, cost-effectiveness could have been improved if a larger share of the funding would have gone to the actual research activities. One indication of cost-effectiveness could be that the original deliverables planned have been produced with less resource inputs than originally estimated. In several projects, especially with large consortia, synergies and joint actions (shared resources, materials) between the partners, have led to serious cost-reductions. Example - QualityLowInputFood: What was really important was that the different partners could call upon the required research facilities, especially the long term field trials. These national/regional field trials are trials where the different partners looked into soil fertility and into the quality of food of completely different farming systems in their own country. The trials are very expensive and are funded for several years (sometimes even more than 30 years) by national money. The different research groups exchanged materials in QLIF, such as soil and plant samples, and compared the results. In addition, some of these trials will be used to investigate new research questions, so this is not only cost efficient, but also durable. Some of the projects, especially where the linkages between the different work packages were quite artificial or even non-existing, could have been more cost-effective if they had been split up in several smaller projects, because of the high coordination costs. There are no economies of scale related to coordination: coordination costs are even perceived to increase exponentially instead of proportionally. In some projects, some of the activities could have been avoided and the corresponding costs could have been saved. Examples are travelling costs related to meetings that could have taken place by video-conferences or telephone. Example - ECNIS: The consortium-partners used a video conferencing system. The equipment was paid back in 1.5 years and the investment should be compared to the savings on travel and time normally spent on attending meetings, but it is expected to be less costly. We conclude that on average, the participants considered the benefits higher than the costs. 59.4% of the respondents indicated the benefits were (much) higher than the costs, while only 8.6% indicated the costs were (much) higher than the benefits. 86

87 8 LONGER TERM PROJECT RESULTS 8.1 Introduction In this chapter, we will give an answer to the following evaluation questions: EQ 26: What has been the effect of the projects on human mobility and training? EQ 27: Has project participation/did project participation have an influence on employment during the project and have those new employees retained employment as a results of the project? EQ 28: How relevant has each project been to pending or ongoing policy legislation, and if so in what way has the project been of influence? EQ 29: Did the projects produce unexpected results? EQ 30: Have the projects obtained any results worthy of exploitation and dissemination? EQ 31: Which of the project ouputs contributed or will most likely contribute to the follow-up and valorisation of the project? The answers on these Evaluation Questions are discussed under the following sections: Influence on human mobility, training and employment Influence on policy Unexpected longer term results Implementation and exploitation of project results 8.2 Influence on human mobility, training and employment The evaluation questions that are answered under this section are focused on the effects of the projects on human mobility, training and employment. In the case studies, we explicitly asked whether participating to the project has/had an influence on human mobility and training. Most of the 30 analysed cases have taken one or more actions, ranging from training courses and workshops to staff exchange and mobility grants. Some of these actions are presented in the following box as an illustration. Box 18: Influence on human mobility and training examples from the cases In BIOCOP, about 12 training events were organized for researchers in the consortium to learn about various techniques or the software developed in the project. The exchange of female researchers within the fellowship scheme also contributed to human mobility. Especially the many young researchers involved in HELENA benefitted from several training activities abroad, exchanges between universities, travelling to all partners organizations involved,. 87

88 In the first year, there were taken no actions in the area of human mobility and training in EURRECA, but currently different initiatives are taken: o o o o o A EURRECA PhD student in one of the partner-universities; Integrated meetings and trainings within these internal meetings; Trainings and courses particularly targeted at younger researchers; Workshops; A staff exchange grant to enhance mobilisation-possibilities. Thanks to GA 2 LEN, on average researchers participated in each allergy school or training course. Each year up to 10 researchers participated in the exchange programme. Participants from both exchange programme and allergy schools recognised that that these events/programmes were excellent in extending the researchers network and giving them access to very senior, renowned colleagues. SABRE had a very good uptake of its mobility and training funds. These were made available to scientists within the consortium, to scientists in NMS or Third Countries and to Early Stage Researchers. Complementary funding of about 2m was obtained by four of the core SABREpartners for a Marie Curie Initial Training Network called SABRETRAIN (MEST-CT ) to fund a multi-partner training of about 7 PhD students and a number of shorter-term mobility placements for young post-doctoral researchers. In the survey, we asked whether project participation has or is expected to have a longer term influence on the knowledge base and the quality of human resources. Here, we made a distinction between the (potential) longer term influences for companies versus the more research oriented organisations, i.e. the higher education institutes and the research organisations. Since human mobility and training are closely related to those two variables, the results of the online survey might be a first indication on how successful the human mobility and training activities were for the improvement of the knowledge base and the quality of human resources. Both companies as well as the research oriented organisations reported, on average, a (tendency to a) large increase on their knowledge base (mean 27 of 3.63 and 3.87 respectively). In all different types of organisations, the influence of project participation on the quality of human resources was, on average, estimated lower. But, where the research oriented organisations reported a small increase with a tendency to a large increase (mean of 3.45), the companies reported only a small increase (mean of 2.96). We also asked companies about the longer term influence of project participation on employment. The mean values for this particular variable are rather low, situated between no change and a small increase. For the large companies, the mean is even closer to no change as compared to a small increase (mean of 2.27). The mean value of the SME-respondents is slightly higher (mean of 2.47). The interview results from the case studies show that in most of the projects the employment in the partner-organisations increased during the project. A more permanent employment, however, is not always assured. This strongly depends on the financial situation and resources of the organisation after the end of the project, and whether or not the research treated under the FP-project is continued within the organisation. Since, if this is the case, it can be very beneficial to keep these researchers employed in order to valorise the experience from the project in other projects. 27 Means with 1= decrease; 2= no change; 3= small increase; 4= large increase; 5= very large increase 88

89 Despite the fact that employment within the consortium-partners was/is not always permanent, many interviewees stated that being involved in an FP-project is always to some extent beneficial for future employment opportunities. The following box provides some examples from the cases with respect to the influence of project participation on employment. Box 19: Influence on employment examples from the cases Some of the researchers working on BioCop moved on to other projects within the organization or got a permanent contract. In QUALITYLOWINPUTFOOD, some of the organisations employed new researchers during the project. For some of these researchers, the QLIF-project led to a more permanent job in these institutes. But also outside the consortium, employment has increased. The organic food sector has grown, with a growing interest of SMEs and research organisations to employ experts in this particular science field. Therefore, the individual skills and knowledge achieved by participating in the QLIF-project were very important to find a new interesting job in the sector. One of the organisations of the SABRE-project employed addititional staff, whose employment has been prolonged via FP7. The project has resulted in contractual employment for a number of scientists during the project. The industry association started out with one half-time staff attached to the project, now it employs 2½ staff to take care of general topics in genomics. In other words, EADGENE has contributed to demonstrating to breeders and industry the importance of the topic and raised awareness among this group of important stakeholders. Most of the 30 analysed cases have taken one or more actions related to human mobility and training, ranging from training courses and workshops to staff exchange and mobility grants. In the survey, both companies as well as the research oriented organisations reported, on average, a (tendency to a) large increase of their knowledge base. The influence of project participation on the quality of human resources was, on average, estimated lower, in particular by the company respondents. The longer term influence of project participation on employment within the companies is limited, according to the survey respondents. The interview results from the case studies show that in most of the projects the employment in the partner-organisations increased during the project. But a more permanent employment, however, is not always assured. 8.3 Influence on policy We also asked the survey respondents to evaluate the contribution of their project to the strengthening of Community food quality and safety policy. The overall mean score 28 was 3.31, which corresponds to a moderate to a slightly high contribution. A more profound answer was provided by the case studies. Half of the projects provided some evidence of influence on pending or ongoing policy, with IPs seeming to have relatively more influence on policy compared to NoEs and STREPs. 28 1= No or very low contribution, 2= Low contribution, 3= Moderate contribution, 4= High contribution, 5= Very high contribution. 89

90 The direct influence on policy, however, is limited. A frequently cited argument is that if the project is still ongoing or has only recently ended, it is far too early to assess a real direct influence on policy. Other arguments are a too limited interaction with policy makers before, during and after the project, or a lack of interest from policy makers. Further, the implementation by policy makers of very basic research is much more difficult than of more practical and applied results. The most common type of influence on the national level is the involvement in policy bodies & advisory boards and strategic action plans. The most common type of influence on the EU-level (especially on DG SANCO) are: harmonization and adaptations of EU-wide applicable standards; development of codes of conduct & protocols; the establishment of European -wide definitions; testing rules & regulations; providing support for evidence-based policy through information & datasharing; refinement of the policy agenda; steering of debates by advice and recommendations; involvement in policy bodies and advisory boards In addition, some projects have an indirect influence on policy via EFSA (policy dialogue, incorporation of results in guidelines,...) and the CIAA. The following box presents some examples of influence on policy from the cases. Box 20: Influence on policy examples from the cases EURRECA has a direct impact on policy legislation, since different EURRECA-partners are involved in several nutrient requirement setting bodies (e.g. Professor Sue Fairweather-Tait in SACN and Professor Pieter van t Veer in the Dutch Health Council). Further, the Serbian partner was invited to write the nutrition action plan for Serbia until 2020 (strategy for nutrition). The EURRECA recommendations are incorporated in this plan that will certainly contribute to the national Serbian legislation. The project will also have an indirect impact on policy legislation via EFSA: the EURRECAnetwork has no legal responsibility to claim the enforcement of the recommendations, but EFSA has. The only uncertainty is to which extent EFSA will actually realise these enforcements. Further, there is an Integrated Activity that is specifically targeted at policy implications. So, having an influence on policy has been an important issue from the start of the project. The processing of food is not very well regulated at the European level. In the QUALITYLOWINPUTFOOD-project (QLIF) a code of conduct was developed, that was implemented to compensate for this lack of regulation. The most direct impact of QLIF was therefore on EU regulation. The implementation of regulation has definitely improved. Some of the recommendations made under the QLIF-project have already been used and will be further updated in the future. Further, they also developed a lot of protocols for high quality and safety organic production, used by companies, farmers and private farmer associations. These protocols were necessary in order to avoid safety problems. TRUEFOOD has an influence on pending or ongoing policy legislation since it is one of the main goals of the federation to lobby on policy legislation. They have a lot of results that could go in new legislation, like about food hygiene packaging, consumer information regulations (DG Sanco) and health claim regulations. The project results on reducing salt in ham without losing conservative power made it to an EFSA regulation (concerning health claims). Research done in ACE-ART led to the adoption of EFSA guidelines on antibiotics use in food and feed. This end result was particularly important as the project aimed to support EU policy in the field of reducing risks related to the use of antibiotics in the food chain. Implicitly, ACE-ART demonstrated that DG Research and academic institutions are together able to support policy in a very delicate field such as food safety. EUROPREVALL has an impact on existing legislation by providing the list of sources of food allergen to policy maker in order to label food. Threshold data will also be provided, although it will take a bit longer (for example how many grams of peanut is tolerable for a majority of peanut allergic 90

91 patients., etc..). DG SANCO is keen to use these data for evidence-based policy. In addition, one of the research centers is taking part in the EFSA. As to policy and legislation SEAFOODPLUS has affected DG SANCO and Traceability standards. HEALTHGRAIN works on a European definition of whole grain which is expected to influence European policy and standards. Currently, the definition is broadly accepted in the European research community and the US also applies it. HEALTHGRAIN plans to present an opinion paper on the overall topic of whole grains and health. Finally, HEALTHGRAIN work contributed to many national projects, and these national projects include the industrial sector. In this manner, HEALTHGRAIN results are widely introduced, and a large impact of the project can be expected. Half of the case study projects provided some evidence of influence on pending or ongoing policy, with IPs seeming to have relatively more influence on policy compared to NoEs and STREPs. The direct influence on policy, however, is limited. The most common type of influence on the national level is the involvement in policy bodies & advisory boards and strategic action plans. In addition, some projects have an indirect influence on policy via EFSA (policy dialogue, incorporation of results in guidelines,...) and the CIAA. 8.4 Unexpected longer term results In what follows we have clustered the answers around the most prevailing unintended results, provided by the survey-respondents. Many of the described unintended results relate to the extension of the networks of the partners, also outside the project team as such, with positive consequences like new research projects, under FP7 but also in other contexts. A number of respondents also described more structural cooperation/cluster initiatives that later on resulted from the network in the context of the project they participated in. Very interesting are also the unintended results that relate to the changes in the education process and curricula. Another unintended result is the improved project management experience. Finally, some respondents described unintended economic results. 8.5 Implementation and exploitation of project results We start with presenting the results of the online survey on the opinions of the respondents on the relevance of the project output for the organisations in view of intended follow-up to and valorisation of the project s results (see Table 8 1). 91

92 Table 8 1: Relevance of project outputs (a) by instrument IP NOE STREP New, improved or integrated tools, techniques, models ,74 3,88 New or improved products/services ,44 3,62 New or improved production processes ,00 3,34 Demonstrators, prototypes ,08 3,09 New, improved or integrated facilities/infrastructure ,74 3,27 Patents or other forms of IPR ,63 3,38 (a) How relevant were the following specific projects outputs for your organisation in view of intended follow-up to and valorisation of the project s results? Means with 1 = very low relevance; 2 = low relevance; 3 = neutral; 4 = high relevance; 5 = very high relevance. Source: IDEA Consult on the basis of survey results Overall, the results show a neutral relevance of the more industry-oriented outputs for the follow-up and valorisation of the project s results: Patents or other forms of IPR get the lowest score, except for the STREPS; Similarly, demonstrators and prototypes get among the lowest scores; Only new, improved or integrated tools, techniques, models score better with a tendency to high relevance for all three types of instruments. In the case studies, we asked the interviewees which of the project outputs contributed or will most likely contribute to the follow-up and valorisation of the project's results. Scientific papers and publications in refereed journals contributed most, followed by new, improved or integrated tools, techniques and models, new or improved codes standards and/or protocols, and new, improved or integrated information systems for data-sharing. The more applied outputs such as demonstrators, prototypes, patents and new products are considered to have a lower contribution to the follow-up and valorisation compared to the more basic scientific oriented outputs, which confirms the survey results. This might be explained by the fact that not all projects are oriented towards more applied research. Some projects are even very basic research oriented. Publications in refereed journals are very important for the worldwide acknowledgement of being experts in the field. However, publications are not always the best communication means towards the industrial community and consumers, because of the too academic language. For these groups, outputs such as leaflets and posters with only the core results and written in less academic language, preferably in the country language, are more effective for the follow-up, valorisation and implementation of the project results. Further, we also asked in the case study interviews whether these project outputs are considered to be exploitable (i.e. whether they can be introduced in the market), whether the project results are/most likely will be developed by industrial partners and whether the project results are/will most likely be licensed out. The project outputs that are considerd to be most exploitable are new, improved or integrated tools, techniques, models and information systems for data-sharing, new or improved codes, standards and protocols and demonstrators/prototypes. Regarding the (potential) development by industrial partners and the (potential) licensing out of the project results, there is far more uncertainty. Only a very limited number of project results are already or are expected to be developed by industrial partners. These are mainly the results of scientific papers/publications 92

93 and of the new, improved or integrated tools/techniques/models. In addition, only 8 of the projects delivered results that are/will most likely be licensed out. In terms of potential economic impact, the experts gave an overall medium level assessment of the cases. In many projects there is potential for an economic impact. However these require some additional work that is dependent on additional resources in order to be commercialised. The assessment of economic potential varies across the types of projects. The IPs have the highest potential for economic impact well above medium ranking, followed by the NOEs that are slightly below a medium ranking. This conclusion also follows from the inherent difference between these instruments, where IPs are more focused on the research results and NoEs have more focus on the network activities. Lowest are the STREPs, where the potential economic impact is low. Generally, the assessment of potential economic impact uses the whole range from very low to very high potential and therefore it must be underlined that this assessment is very specific for each project. Some cases do not have, and are not expected to have, a potential economic impact, while for others it is their prime goal. We conclude that scientific papers and publications in refereed journals are most likely to contribute to the follow-up and valorisation of the project results. The more applied outputs such as demonstrators, prototypes, patents and new products have a lower contribution to the follow-up and valorisation compared to the more basic scientific oriented outputs. There is uncertainty regarding the (potential) development by industrial partners and the (potential) licensing out of the project results. Only a very limited number of project results are already or are expected to be developed by industrial partners. In terms of potential economic impact, the experts also gave an overall medium level assessment of the cases. In many projects there is potential for an economic impact. However these require some additional work that is dependent on additional resources 93

94 9 CONTRIBUTIONS TO THE STRATEGIC OBJECTIVES 9.1 Introduction This chapter deals with the following evaluation questions: EQ 32 Have overall programme objectives been met? EQ 33 Did the activities contribute to developing a European Research Area? EQ 34 Has there been any impact on European competitiveness and scientific performance through increased knowledge, capacities and infrastructures? EQ 35 What have been the effects of the programme on Community social objectives such as quality of life, health, employment, the environment? These questions address the degree to which the projects have contributed towards meeting the strategic objectives of the FP6 TP5. The answers to the above evaluation questions are discussed under the following headings: Overall contributions towards strategic impact Contributions towards establishing the European Research Area Contributions towards enhancing European competitiveness and scientific performance Contributions towards meeting Community social objectives However, before presenting the findings it is essential to discuss in general terms the nature of such impacts, the difficulties associated with measuring impact at a time where many projects have finished quite recently or are still ongoing, and the approach to measuring impact in this evaluation (cfr. the caveat paragraph in the section on the evaluation methodology and tools). Strategic impact is a measure of the broader consequences of projects such as technical, economic, social, political or environmental effects on those who have already benefitted up to now but not least potential future beneficiaries be it locally, nationally or at EU level. There are two central challenges to assessing impact. First: boundary judgment, i.e. deciding what effects to select for consideration. Second: attribution, i.e. what effect is due to what. Because, on the one hand, effects can be numerous and varied, and on the other hand they are typically the result of complex interactions, assessing impact is difficult in most circumstances. In the majority of cases, it is difficult to attribute rigorously broad effects on different beneficiary groups and at different levels over time to a specific research project or programme. While a few impacts occur in the short term, most impacts only materialise in the medium to longer term, i.e. maybe three, five or even ten years after a project has completed. As mentioned before, the TP5 projects subject to this evaluation have only ended recently or are still ongoing and therefore the bulk of the impacts have not materialised yet. Ideally, strategic impact should be identified as perceived or experienced by all present and potential future beneficiaries and 94

95 users. In this evaluation we have for methodological and practical reasons opted to obtain the views of the project participants on realised and potential impacts as it has been outside the scope of the evaluation to involve potential external and future beneficiaries more broadly. The risk of this approach is that the impact measurements or scores provided by the repondents may tend to overestimate the potential future impacts for two reasons. One is that project participants are likely to be loyal to their respective projects, too optimistic about the impact of the research that they have contributed to, and generally not the best positioned to assess likely future impact. The other reason is that it appears that follow-up efforts to maximise potential strategic impacts will be quite limited in the future. For example, there are no EC FP6 programme funds or national resources set aside for this purpose. Some projects have established project associations or other types of project successors to make outputs and results more sustainable but it is difficult for the partners to judge at this point to what degree these follow-on entities will become effective and how long they will last. The realised and expected impacts described below should be viewed against this background. 9.2 Overall contributions towards strategic impact Findings Have the overall or strategic - programme objectives been met? A first answer to this key evaluation question is provided by the results of the survey. The participants in the survey were asked about the contribution of their project towards the strategic objectives of Thematic Priority 5 Food quality and safety. The proposed potential impacts that the respondents were asked to relate to was inspired by the hierarchy of objectives and effects developed in the first stage of the evaluation. We made a distinction between (i) scientific and technological impacts on knowledge producers, (ii) impact on diffusion and exploitation in terms of knowledge transfer and intellectual property, and (iii) socio-economic impacts. In Table 9 1 we have listed all the potential contributions of the projects towards meeting the strategic objectives that the respondents were asked to address. The impacts are presented according to their mean scores in terms of already realised or potential contribution of the projects to meeting these objectives. Except for one of the strategic objectives, which is ranked lowest with a value well below 3, projects were judged by the respondents to have provided a moderate (value 3) up to a high (value 4) contribution to meeting each of the objectives in this list. However, all values remain well below the value 4 = high contribution. Only the first four of the ranked objectives get a mean score above 3.5. The first two ranked objectives relate to the impact of the projects on diffusion and exploitation in terms of knowledge transfer and intellectual property. Overall, the projects contributed most to the realisation of the following objectives: Exchange of knowledge and expertise in the area of food quality and safety with research partners across the world; Improved collaboration and coordination in the area of food quality and safety. 95

96 The third and fourth objectives relate to the scientific and technological impact on knowledge producers and users: The development of scientific and technical excellence in the area of food quality and safety; The strengthening of the scientific and technological bases of the Community food quality and safety industry. Table 9 1: Contribution of projects towards achieving strategic objectives as perceived by project participants Category of strategic objective Strategic objective Mean % No opinion Knowledge Transfer Knowledge Transfer Scientific and Technological Scientific and Technological Exchange of knowledge and expertise in the area of food quality and safety with research partners across the world Improved collaboration and coordination in the area of food quality and safety The development of scientific and technical excellence in the area of food quality and safety The strengthening of the scientific and technological bases of the Community food quality and safety industry % % % % Socio-Economic An improved understanding of the link between food and health % Scientific and Technological Knowledge Transfer Detection of new innovative solutions in the area of food quality and safety Bringing research in the area of food quality and safety closer to society % % Socio-Economic A higher quality of food for European citizens % Knowledge Transfer The strengthening of Community food quality and safety policy % Knowledge Transfer Improved researchers' mobility in the area of food quality and safety % Socio-Economic Improved control of food-related risks % Socio-Economic Improved control of food production % Scientific and Technological Development of new innovative products or services % Socio-Economic Socio-Economic Knowledge Transfer Knowledge Transfer Improved control of environmental factors related to food quality and safety The development of an environmentally friendly production and distribution chain of safer, healthier and more varied food A faster conversion of research results in the area of food quality and safety into marketable innovations Improved research infrastructure in the area of food quality and safety % % % % Socio-Economic Improved control of health risks associated with environmental changes % (a) How would you evaluate the contribution of the project to the following strategic objectives: Socio-economic impact of research activities ( Socio-economic ), scientific and technological impact on knowledge producers and users ( Scientific and Technological ), impact on diffusion and exploitation in terms of knowledge transfer and intellectual property ( Knowledge transfer ). With mean trend: 1= No or very low contribution, 2= Low contribution, 3= Moderate contribution, 4= High contribution, 5= Very high contribution. Source: IDEA Consult on the basis of survey results 96

97 If we rank the three categories of strategic objectives in terms of the overall average realised or anticipated contribution of the projects to meeting these objectives (i.e. the impact of the projects), we get the following results: 5. Scientific and technological impact on knowledge producers and users (overall mean value = 3.47); 6. The impact on diffusion and exploitation in terms of knowledge transfer and intellectual property (overall mean value = 3.39); 7. Socio-economic impact of research activities (overall mean value = 3.14). In other words, the respondents find that the contributions of their projects towards meeting the strategic objectives have been or are likely to be moderate to high (closer to moderate) as an average response to the questions. This could be interpreted as a good contribution. The contributions towards meeting scientific and technological as well as diffusion and exploitation objectives are somewhat higher than to meeting the socio-economic objectives. This is not surprising as socio-economic impacts are of a nature that generally takes time to materialise. But it is also in line with the earlier observations that industrial participation was limited and that industry-relevant outputs were much less prevelant compared to research-relevant outputs. Although these findings cover up divergenses among the types of instruments (NOEs, IPs, STREPs and CAs), types of participants (research oriented organisations or private companies), country groups, and reserach topics (see background report), the general finding is that the contributions to meeting the strategic objectives is good as seen by the project participants. Given a total number of 732 respondents to the survey we consider this a robust conclusion. The eventual contribution may, however, turn out to deviate from this response by the project partcipants for the methodological reasons discussed above. Moreover, the case studies provide additional information to answer this evaluation question. More specifically, the experts have been very positive towards the projects, and all but three 29 projects receive a high or very high overall assessment of their contribution to the strategic and specific objectives of the FP6 TP5. It is notable that the STREPs lie considerably below the average assessment. One reason for this could be found in the size of the projects: as they are not as large as NoEs and IPs, their results are also less substantive. For examples of the overall contributions, the experts mention in several cases that the projects have gathered the scientific world (critical mass) within the particular topics, supported networking, strengthened scientific excellence and that the projects have contributed with knowledge that will support the health and well-being of the European citizens also in the future. In other words, durable integration has happened and the European research area is strengthened. Some projects have also contributed to the political processes, e.g. by providing knowledge to EFSA. Many projects have also contributed to the European competitiveness, e.g. by developing new innovative products. Only few projects have contributed to the third strategic objective of sustainability and preserving the environment. In cases where associations or other durable structures have been established, expectations are high that these will take results forward, fill out any gaps and add value and ensure communication of knowledge within the particular area in the future. 29 All three are STREPs one of these was not assessed due to lack of material. 97

98 9.2.2 Measures to enhance strategic impact There was an open question at the end of the survey questionnaire where respondents could make additional comments on how to improve the impact of the research funded under TP5. We have grouped the survey responses and combined them with the findings from the case studies as follows 30 : Scope of the calls: Some participants find that calls should be more detailed in order to know exactly what the EC would like to support and thereby reduce transaction time and costs. Others find the contents of the call should be less detailed to allow for more innovative ideas and methodologies to meeting programme objectives. Type of research: Some find that the programme should put emphasis on basic research ( real research ) rather than on applied research in order to allow more for unexpected results and not push the researchers into tied assignments. Others find applied research to be more cost-effective in reaching solutions that can contribute to meeting overall programme objectices comparatively faster. Size of projects: Most respondents consider smaller projects to have a comparative advantage in generating results in an efficient way. The risk of making projects too big or all encompassing is that this may reduce or eliminate competition between different smaller research groups in terms of developing alternative, innovative approaches to meeting research objectives. Bigger projects, especially those that develop standards and integrate databases, benefit from many participants, however. Involvement of SMEs: In the view of many respondents involvement of SMEs is important to create a link to industry. Some maintain, though, that they have relatively limited capacity to do so and that they provide limited value added. Follow up on projects: There is a clear need to follow up on the project outputs and results to maximise future impact, e.g. by additional funding support. Otherwise, there is a great risk that large resources used to produce these outputs and results will get significantly lower returns in the form of impact than could otherwise have been achieved. One way could be to prolong project implementation periods to internalise efforts to promote impact. Dissemination: More attention and funding for dissemination are required. Too much focus is put by the projects on producing deliverables. More efforts should be made to increase use and uptake of the deliverables. EC management and procedures: EC management and procedures should be improved to support flexible and cost-effective project implementation allowing the project to change methodological and implementation course when warranted in a quickly changing technology and business environment. 30 Note that sometimes the suggestions that were made by the respondents are opposite to one another. 98

99 9.3 Contributions towards establishing ERA The aim of the European Research Area (ERA) is to enable researchers, research institutions and businesses to increasingly circulate, compete and co-operate across borders thereby giving them access to a Europe-wide open space for knowledge and technologies in which transnational synergies and complementarities are fully exploited. A key evaluation question is if the programme activities have contributed to developing the ERA. Are there in other words any clear indications on emerging wider European structures, policies and programmes visible in the Member States as a result of these projects? Answers to this question can be derived from different sources: Responses to the survey questions regarding: o the relevance and the quality of project outputs underpinning the construction of ERA, e.g. establishment or enlargement of research networks; common strategic action plans; joint/common initiatives; research mobility schemes; new, improved or integrated facilities/infrastructures; etc.; o o o ERA-related longer term impacts on participants; the contribution of each project to strategic objectives concerning the construction of ERA, e.g. improved collaboration and coordination, improved researchers mobility, etc.; the stability and renewal of research consortia across FPs; The analysis of FP participation data, in particular the number of domestic and cross-country links created by the different food projects Project outputs The development of high quality common research infrastructures at a European scale is a key feature of the ERA. The term research infrastructures refers to facilities, resources and related services that are used by the scientific community to conduct top-level research in their respective fields. This definition covers major scientific equipment or set of instruments, knowledge-based resources such as collections, archives or structured scientific information; enabling ICTbased infrastructures such as grid, computing, software and communications; any other entity of a unique nature essential to achieve excellence in research. To this end, the projects have developed outputs that serve to enhance common research infrastructures such as: common new, improved or integrated tools, techniques, models; common new, improved or integrated information systems for data-sharing, including data bases and knowledge repositories; common new or improved codes, standards, protocols and norms; common quality management systems (QMS); guidelines and best practices. 99

100 According to the survey-results that were reported before (and are described more extensively in the background report) the respondents consider such outputs to be of medium to high relevance (mean values between 3.60 and 3.84) for their respective organisations. Further, the respondents regard the quality of the outputs as high (mean values between 3.95 and 4.05). These outputs have contributed to the ERA by providing a common platform for European research carried out already by the projects and for future research in the research fields concerned Longer-term impacts In addition, we also asked the survey participants about their views on the longer term impact of project participation on a number of variables for their organisation. Some of these variables are relevant indicators for the realisation of the ERA. Table 9 2: Longer term impact of project participation for research oriented organisations Mean Total (n=616) % IDK % too early to assess Knowledge base % 1.3% Integration into the international scientific community % 1.9% International scientific standing % 2.4% Quality of human resources % 3.7% (a) What has been (or do you expect to be) the longer term impact of project participation on the following variables for your organisation? Means with 1= decrease; 2= no change; 3= small increase; 4= large increase; 5= very large increase Source: IDEA Consult on the basis of survey results The responses from the research oriented organisations clearly indicate that the projects have already or are likely to contribute quite significantly to especially their knowledge base, integration into the international scientific community and international scientific standing but less so to the quality of their human resources. This response is a firm indication of the projects realised or potential contribution to the ERA as well as high contributions of the projects to their scientific status. Table 9 3: Longer term impacts of project participation for companies Mean Total (n=91) % IDK % too early to assess Knowledge base % 1.1% Number of strategic partnerships % 5.5% Quality of human resources % 2.2% Number of R&D collaboration agreements % 6.6% 100

101 Mean Total (n=91) % IDK % too early to assess Technology base % 2.2% (a) What has been (or do you expect to be) the longer term impact of project participation on the following variables for your company? Means with 1= decrease; 2= no change; 3= small increase; 4= large increase; 5= very large increase Source: IDEA Consult on the basis of survey results The responses from company participants show that their knowledge base has or is expected to increase to a fairly high extent (overall mean value = 3.63) as a result of participating in the FT5 projects. The projects may also contribute to cerating new partnershps, improving the quality of their human resources, research collaboration with others and their technology base. However, such contribution is expected to be somewhat limited Achievement of strategic objectives The following table shows the contribution of projects to meeting a number of strategic objectives that were selected because they represent different aspects of developing the ERA. Table 9 4: Contribution of projects to the development of the ERA (a) Strategic objective Mean % No opinion Exchange of knowledge and expertise in the area of food quality and safety with research partners across the world % Improved collaboration and coordination in the area of food quality and safety % Improved researchers' mobility in the area of food quality and safety % Improved research infrastructure in the area of food quality and safety % (a) How would you evaluate the contribution of the project to the following strategic objectives. With mean trend: 1= No or very low contribution, 2= Low contribution, 3= Moderate contribution, 4= High contribution, 5= Very high contribution. Source: IDEA Consult on the basis of survey results According to the respondents the projects have or are likely to provide moderate (value 3) to high (value 4) contributions to these strategic objectives. Especially the exchange of knowledge and expertise and improved collaboration and coordination score quite high in the projects. The box below contains examples of projects that have or are likely to contribute successfully to ERA-related objectives. Box 21: Contributions towards establishing ERA - examples from case studies CO-EXTRA (IP). The strategic objective of the project was to provide stakeholders in the food and feed chains with tools and methods allowing them to practically implement co-existence and traceability of GM and non-gm products as required by EC regulations and demanded by European consumers. CO-EXTRA developed cost-effective and reliable tools for the co-existence and traceability of GM conventional and organic crops. The design of such tools had to consider gene flow management, costs and methods of segregating GM and non-gm products, GMO sampling and detection, and liability and compensation. The results were integrated into an electronic Decision Support System (DSS) available to stakeholders who wish to determine the practical and economic aspects of co-existence and traceability. 101

102 CO-EXTRA has contributed to reinforce the European Research Area through strong collaboration with existing European centres, networks and projects as well as several third countries. Several laboratories of the European Network of GMO Laboratories (ENGL) are among its members. The project was primarily meant to provide scientific basis for political decisions. However, the tools developed were mainly aimed at government enforcement laboratories and were transferred to them through the ENGL network. Some research results have been converted into marketable products during the project. In other cases, laboratories and companies are still investigating how to exploit the results. One research centre signed a contract with a company outside the CO-EXTRA consortium to develop methods for allergen detection. Some companies do market innovations from the project. It has been the policy of CO-EXTRA to make most of its results available free for public and private laboratories and companies. TRUEFOOD (IP). The overall aim of TRUEFOOD was to introduce suitable innovations into traditional food industry to maintain and/or increase the competitiveness of the industry in an increasingly global European market place. The project was coordinated by the European Economic Interest Group SPES GEIE comprising 11 European Food and Drink Federations. The above aim has been achieved through close integration of research, demonstration activities, and training and dissemination efforts. It is certain that solutions developed in the context of the project have been implemented in 1,700 companies, another 6,800 companies are quite likely to have implemented new solutions, but this cannot be verified, and an additional 23,500 companies have become more sensitive to innovation, at least according to the interviewed participants. The project is firmly believed to have important long-term and sustainable impacts, e.g. by Technology Development Units within the national federations. Attention is now devoted by traditional SME food producers to innovation as a very important outcome. There will also be impacts on pending or ongoing policy legislation since it is one of the main goals of the federation to influence legislation. Thus, TRUEFOOD has produced many results that can inform new legislation on topics like food hygiene, packaging, consumer information regulations (DG Sanco) and health claim regulations. The project results on reducing salt in ham without losing preservative effect made it to an EFSA regulation concerning health claims. BIOTRACER (IP). The aim of the project has been to make a multi-disciplinary attempt to trace the course of food and feed contamination, using the latest metabolomic and genomic data to understand the physiology of microorganisms. Three food chains were analysed and domain models built that are now available to users. BIOTRACER will not have an impact on consumers immediately. It deals with policy and standards for the EU and national governments addressing e.g. contamination models and slaughterhouses. Although the project has established a link between research and food production, it is unlikely to bring products to the market in a faster way than today in the short term by using the rapid diagnostic methods and mathematical models developed. This impact may only materialise in 5-10 years. BIOTRACER has put traceability on the map and developed a strong nucleus around this research topic in Europe. After its termination, the project is planned to be developed into a so-called BIOTRACER Community whose activities will centre on tracking and tracing pathogens in food and feed chains. Newly developed laboratory protocols and strains of pathogens can be shared among the researchers/members and biotracing certification based on a standard written by the BIOTRACER Community may be introduced. If and when these plans are realised, BIOTRACER will become a repository of relevant information in this research field Cross-country links Looking at domestic and cross-border linkages between the project participants the following picture emerges. Figure 6: Distribution of domestic links by country 102

103 Source: Idea Consult on the basis of the FP6 Dbase and updated information from project coordinators The 181 food projects generate 43,835 links, of which 3,334 are domestic links and 40,251 cross-border links. A domestic link is defined as collaboration within a project between two partners in the same country, while a cross-border link denotes collaboration within a project between two partners in different countries. Figure 5 shows the distribution of the domestic links among the different participating countries. UK is the country with most domestic links, representing 21% of the total. Remarkable is also the Netherlands, one of the smaller Western European countries, that has the second largest number of links equal to 13% of the total. Six of the eight countries with the highest number of domestic links are Western European. The table below shows the distribution of the 43,585 links, currently generated by TP5, in a country-group matrix. There are 25,890 country-group links in total (dark grey shaded areas). These are links between participants belonging to the same country group. The remaining 17,695 are cross-country group links, i.e. links between two participants from different country groups. As expected, most links are to be found within the group of old member states. Most cross-country group links between participants from the old member states are with participants from the new member states (5,353 links), the associate states (3,949 links), the candidate countries (1,214 links) and the INCO Latin American countries (1,134 links). The large number of domestic and in particular cross-border linkages generated by the projects is an important indication to the effect that Priority 5 has contributed to the realisation of ERA. 103