Evaluator's view. Riitta Niemelä Senior Lecturer, Environmental Technology Vaasan ammattikorkeakoulu

Transcription

1 Evaluator's view Riitta Niemelä Senior Lecturer, Environmental Technology Vaasan ammattikorkeakoulu

2 Vocabulary REA Research Executive Agency IAR Individual Assessment Report CR Consensus Report VC Vice Chair IO Independent Observer Remote Evaluation Central Evaluation Consensus Meeting

3 Marie Curie Evaluation Process A high number of proposals free choice of research topics relatively small projects Huge number of proposals on all possible topics!! Proposals are sorted -> different panels (ECO-SOC, CHE, LIFE, ENV, ) Evaluation has two phases: remote and central

4 Who is an Evaluator? Anyone can register in the FP database to become an evaluator The REA staff picks up suitable persons from the database Professional experience Academic/private sector Gender balance, geography???

5 Why to Work as an Evaluator? It is interesting! A possibility to read front edge proposals Networking Money Private Employer

6 Marie Curie Evaluation Variety of themes High number of small proposals The number of evaluators is limited -> Global & General Experts??? The persons evaluating the proposal are possibly (probably) not experts in exactly your field.

7 Evaluator Is asked about availability (remote & central) Might be asked to participate Has no conflict of interest Is given ~ 10 (20) proposals to evaluate Performs the remote evaluation Participates in the central phase Writes consensus reports Final panel meeting with the ranking list After that something happens that is not our business!

8 Evaluators of a Proposal In Marie Curie usually 3 evaluators Experts / Generalists Different organisations and countries

9 Evaluation Criteria in Marie Curie Science & Technology Training / Transfer of Knowledge Researcher - for the individual grants Implementation Impact Subcriteria for each of these

10 S&T 1 Easy criterion Research/technological quality, interdisciplinary and multidisciplinary aspects Originality and innovative, relationship to the 'state of the art' of research in the field Timeliness and relevance of the project Expertise/quality of the host/group/supervisors

11 S&T 2 Everything you really wanted to tell about the project All essential info has to be in the proposal To remember about the evaluators: Not everyone has deep expertise exactly in this field Most are researchers. They like to be given literature references. Are not expected to look for additional information (e.g. web links).

12 Training / Transfer of Knowledge 1 What is going to be done? Great science alone is not enough Subcritetia for example: Clarity and quality of objectives Experience of the host in training Transferable skills Exposure to private sector

13 Training / Transfer of Knowledge 2 Clear plans Additional/transferable skills Capacities/experience of the organisation The evaluators take nothing for granted. Even the largest universities have to demonstrate their capacity. Is private sector involvement required/recommended?

14 Implementation 1 Project management Infrastructure / facilities and international collaboration Practical arrangements for the implementation and management of the project Feasibility and credibility of the project + work plan

15 Implementation 2 Private sector required? Be careful with artificial partners Keep the plan simple and clear! Quite easy criterion for the evaluator even not being an expert in the field

16 Impact 1 Here you should put some effort! Maybe the most difficult criterion Contribution to European excellence and European competitiveness Benefit to the European Research Area. Impact of proposed outreach activities (dissemination) Potential of. having some impact on the participants or in a wider European context

17 Impact 2 Good science = great impact? Poor science. Is it possible to achieve good impact? This criterion makes ~20% of your score! Here you can make the life easy for the evaluators

18 Scoring 0 - The proposal fails to address the criterion under examination or cannot be judged due to missing or incomplete information; 1 - Poor. The criterion is addressed in an inadequate manner, or there are serious inherent weaknesses; 2 - Fair. While the proposal broadly addresses the criterion, there are significant weaknesses; 3 - Good. The proposal addresses the criterion well, although improvements would be necessary; 4 - Very Good. The proposal addresses the criterion very well, although certain improvements are still possible; 5 - Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor.

19 Remote Evaluation IARs 10/20 proposals in pdf to read and evaluate in 2 weeks Work at home or office, day or night Hundreds of pages on obscure themes to read in no time Produce Individual Assessment Report (IAR) with scores for subcriteria Load the report into the system Work in isolation! No info on the other evaluators identity or opinions Not contacting the proposers!

20 How? (my way of doing it) Remote Evaluation, IARs Print out all the proposals Check the call & evaluation criteria See the briefing by the commission online Read a few proposals to get some touch Start reading thoroughly taking notes for the report Write a few IARs Load the first IARs into the system Continue with all the remaining proposals

21 How? Each proposal takes hours to read and evaluate Max 2 /evening 3-4 /day during weekends Difficulties to fall asleep? Become an evaluator!

22 Remote Evaluation, CRs Consensus Report (CR) is the final outcome of the evaluation A few days before the central phase Rapporteur tasks are assigned The IARs of other evaluators (3) become available The rapporteur can start preparing a draft CR Not obligatory

23 How? Remote Evaluation, IARs Check if the 3 evaluators agree more or less Analyse if I missed something For the ones that I am Rapporteur: Collect comments Sort them Check the proposed scores

24 Central Evaluation All evaluators together in Brussels 2-5 days Consensus meetings Preparing final CRs Office hours evening work Vice Chairs Moderate in consensus meetings if the evaluators have difficulties to find consensus Check the CRs Do not read the proposals

25 Consensus Meetings The 3 evaluators participate Busy days! 30 min to discuss the content of the CR and agree the scores Usually go smoothly VC may moderate in case of disagreement Sometimes extra time required

26 No Consensus? Sometimes the evaluators cannot reach consensus Possible to involve a fourth evaluator and try again with more views

27 Consensus Reports 1 Rapporteur writes Format: Strengts Weaknesses Overall comments Other 2 evaluators check and approve text and scores Vice Chair checks quality and language Scores and wording: Don t say very good and give 3 Don t give 4 or lower with no weaknesses

28 Consensus Reports 2 Should be informative Should not give advice on how to improve the proposal The proposer should be happy with the report even if not being funded (say REA staff) Redress possibility for the proposers if they are not happy with the results of evaluation To avoid redress the CRs are very neutral

29 Consensus Reports 3 Corrections after VCs comments Each evaluator approves the CR in the system Final version printed out nd signed by the Rapporteur and VC CR is the feedback the proposer gets

30 Final Panel Meeting All proposals in the panel are listed according to their total score Scores of individual criteria with weighing Final check for possible errors in scores Clean your desk and computer and go home

31 Does It Work? Surprisingly well In my opinion the evaluation process is fair Really good proposals are found and funded also the really bad ones Is the process cost-effective????

32 A Good Proposal? Contains answers to all evaluation criteria Information can be found under each criterion Recommended font size Does not exceed page limitation References in the S&T section Check the consistency Not copy-paste of old proposals Keeps the evaluator awake when reading