2015 Research Trainee Program Competition for Post-Doctoral Fellowship Awards EVALUATION CRITERIA FOR REVIEWERS

Transcription

1 2015 Research Trainee Program Competition for Post-Doctoral Fellowship Awards EVALUATION CRITERIA FOR REVIEWERS VERSION 1 LAST UPDATED: JANUARY 28, 2015

2 Note to ALL Reviewers: Read all assigned applications before rating any of them. Please consider when rating: o Career interruptions for child bearing and raising can affect publications and other related knowledge production. o Different disciplines and environments offer different opportunity for publication. o Reputation of host institutions should not affect your evaluation of the applicant or their research training environment. o Publications: journal impact factors vary from discipline to discipline and do not necessarily indicate the quality of the individual publications. Health professionals: Health professional applicants (MD, DDS, DVM, DPharm) will have a very different (but still research-appropriate) track record from applicants in the PhD career track. Please note that the weightings for two of the evaluation criteria have been adjusted to ensure that the health professionals remain competitive with applicants from the PhD career track. The weightings have been adjusted for the research project and publications criteria. Knowledge translation: Applicants across all research themes have been encouraged to include a range of appropriate knowledge translation (KT) activities, from traditional activities such as conference presentations and publications, to non-traditional activities such as development and implementation of toolkits; educational sessions with patients, practitioners, or policy makers; media engagement; or commercialization of scientific discoveries, in their research project, where applicable. Notes: Applicants may submit research projects that are fully focused on KT science; these will be evaluated against specific KT science criteria. KT science relates to exploring the theories, mechanisms, concepts and/or methods by which evidence is used in healthcare practice and policy. Each application must be rated in each of the six evaluation areas: o Research project o Training environment o Research activities and related research achievements o Career development o Reference letters KT may be included in research projects in one of two ways: o As components of the research project (the practice of KT) o As the focus of the project (KT science) All applications have been reviewed by MSFHR Programs staff and are eligible for the 2015 Research Trainee Post-Doctoral Fellowships competition. MSFHR 2015 Research Trainee Competition: Evaluation Criteria for Reviewers Page 2

3 Evaluating the Research Project Post-doctoral fellows from all research themes including health professionals with a PhD Health professionals without PhD, pursuing degree or research only Weighting 25% 35% Assessment Criteria Consider the overall grantsmanship and the scientific merit of the proposal. What is the potential impact and reach? How novel is the research project? If not novel, to what extent does potential impact compensate? Has a reasonable hypothesis/research question been generated? Is the research likely to produce new data and concepts or confirm existing hypotheses? Is there clarity in the rationale for the research approach and methodology? Is the feasibility of the research approach clear (including recruitment of subjects, project timeline, and preliminary data where appropriate, etc.)? Is the project the best fit for the applicant given their education, experience and interests? Is there a plan for disseminating the research results (i.e. planned traditional and/or nontraditional KT activities)? Is there a plan for implementing the research results, if appropriate? Additional criteria for KT science projects only: o Does the proposal clearly describe how the dissemination or implementation of research findings is being studied (e.g. objectives, proposed KT activities, rationale, intended impacts, KT approach/methodology, identification and engagement of target audiences)? o Does the project address the study of both the process and the outcomes of the KT research? o Will the project add to the body of knowledge on effective KT (e.g. KT models and mechanisms, methodological questions in conducting KT research, developing new KT models, and/or the effectiveness of processes to develop more effective KT strategies)? Assessment Descriptor Outstanding The project embodies leading edge concepts. The description of the hypothesis and methodology are clearly stated and are of the calibre expected of a post-doctoral fellow who is at the forefront of the field. Potential impact of the project is explained clearly and plans for dissemination and implementation, if appropriate, are exemplary. The project is an ideal fit to the applicant s education and research experience. Score MSFHR 2015 Research Trainee Competition: Evaluation Criteria for Reviewers Page 3

4 Excellent The project has the potential to contribute substantially to knowledge in this field. The description is clear and concise, providing enough methodological detail to suggest that the goals of the project will, in all likelihood, be achieved. Dissemination and implementation activities, if appropriate, are clearly explained. The project is a highly suitable fit to the applicant s education and research experience. Very Good The project will offer a challenging training program for the applicant, and will add to the existing knowledge in this area of research. It provides sufficient rationale and methodological detail. Potential impacts of the project plan and dissemination and implementation activities, if appropriate, are appropriately presented. The project is a very suitable fit to the applicant s education and research experience. Good The project will offer good training for the applicant and will add to the existing knowledge in this area of research. The project information is present but not well integrated. It is likely that the research will have some impact. The dissemination and implementation activities, if appropriate, are satisfactory. The project is a suitable fit to the applicant s education and research experience Adequate Below Adequate Not Acceptable MSFHR 2015 Research Trainee Competition: Evaluation Criteria for Reviewers Page 4

5 Evaluating the Training Environment Weighting 20% Assessment Criteria How well is the supervisor(s) funded? Does he/she have consistent peer-reviewed funding? Review the grants currently held, noting the extent to which the supervisor is either listed as a principal or co-applicant for the funds. Is there a fit between the applicant and the supervisor(s); does the proposed project relate to the supervisor s overall research program? Are the research methods used by the supervisor(s) congruent with the applicant s? Does the supervisor s publication record provide a sense of productivity, impact and collaboration? Consider the different disciplines and their impacts when evaluating the publication record. Does the supervisor(s) offer an intellectually challenging and supportive environment where the applicant will have exposure to other trainees, facilities, equipment and personnel support? Review the training record and evaluate the level of training, length of time with supervisor, degree received (if applicable), and current position. Keep the career stage and discipline of the supervisor in mind as your expectations of mentoring by a recently-established investigator should differ from a long-established researcher. Additional criteria for KT science projects only: o Does the supervisor hold KT science funding? o Does the supervisor(s) publication record demonstrate a clear focus on KT science? Assessment Descriptor Outstanding The peer-reviewed funding that the supervisor has received and continues to receive is outstanding and has established first-class facilities. The level of training offered is exceptional in terms of potential interactions with other researchers/trainees. This training environment provides for significant opportunities for contributions by the trainee as first author on quality peerreviewed publications. Fit between the supervisor s area of research and the applicant s research project is solid. Outstanding mentorship. Excellent The supervisor currently receives strong peer-reviewed funding from recognized agencies and the trainee will interact with numerous researchers/trainees who appear to receive strong mentoring. Fit between the supervisor s area of research and the applicant s research project is very good. Excellent mentorship. Score MSFHR 2015 Research Trainee Competition: Evaluation Criteria for Reviewers Page 5

6 Very Good The supervisor s projects are well funded by peer-review based agencies providing the applicant access to the appropriate resources within the immediate environment. The environment also includes a number of trainees having various degrees of research experience and track records of publications. Fit between the supervisor s area of research and the applicant s research is strong. Good The supervisor has funded projects by peer-review based agencies providing the applicant access to some resources within the immediate environment. The environment is suitable for the applicant and there is a good fit between the supervisor s area of research and the applicant s research project. Appropriate mentorship Adequate Less Than Adequate Not Acceptable MSFHR 2015 Research Trainee Competition: Evaluation Criteria for Reviewers Page 6

7 Evaluating Related Research Activities and Other Achievements Weighting 5% Assessment Criteria Consider the career path that applicants have followed to date. Assess the number, importance and breadth of the applicant s special achievements relative to their education and experience. Consider indications of the applicant s activities and most significant contributions and interest in research activity including distinctions, awards (applied for and/or received) and credentials, presentations (regional, national and international), KT activities, and other academic and nonacademic achievements). Note the length of time required to complete academic programs. For their level of training, has the applicant shown professional leadership and accomplishments in research and research related activities including KT activities? Has the applicant shown an interest in research through the development of other skills and additional research activity? Has the applicant worked in another research lab or research environment? Additional criteria for KT science projects only: o Other achievements may include other non-traditional methods of dissemination such as the development and implementation of toolkits, blogs, guidelines, books, tutorials, podcasts, videos and e-discussions. Alternative methods of disseminating research findings should be considered and weighed appropriately. Assessment Descriptor Outstanding Applicant is involved in an exceptional number of other research activities. All aspects of the applicant s distinctions (national/international presentations, awards, credentials, funding awards, KT, and other academic and non-academic achievements) for his/her stage of research career indicate recognition of an outstanding level of talent. Applicant s strong, original contributions to knowledge are widely recognized. Excellent Applicant has involvement in a significant number of other research activities including a number of presentations (including national/international), distinctions, awards, credentials, funding awards, KT, and other academic and non-academic achievements that demonstrate a level of excellence for his/her stage of research career. Applicant s strong, original contributions to knowledge are well recognized. Several aspects of the applicant s activities and distinctions indicate excellent level of talent. Score MSFHR 2015 Research Trainee Competition: Evaluation Criteria for Reviewers Page 7

8 Very Good Applicant has involvement in a number of other research activities including presentations (regional/national), distinctions, awards, credentials, funding awards, KT, and other academic and non-academic achievements that indicate a very good level of talent. Good Applicant has other research experience, but has yet to participate in formal presentations. Has received limited awards (related to awards in support of research) and is generally less competitive in nature. The applicant s activities and distinctions indicate an average level of talent Adequate Less Than Adequate Not Acceptable MSFHR 2015 Research Trainee Competition: Evaluation Criteria for Reviewers Page 8

9 Evaluating Research Goals and Career Development Weighting 10% Assessment Criteria Does the project complement the applicant s research career pathway? Has the applicant exhibited diversity in their career path (e.g. pursuing research opportunities at different affiliations/organizations)? Is there clarity and logic in the explanation of the applicant s plan for a research career? What is the pace of the progress of the applicant and relevance of the proposed training? What is the potential of the individual to become an independent investigator? Assessment Descriptor Outstanding Outstanding track record at every step of research pathway. Progressing at exceptional pace and flawless depiction of research career goals and relevance of the proposed training. Ideal career path. Excellent Excellent track record at every step of research pathway. Progressing at first-rate pace and clear and convincing depiction of research career goals and relevance of the proposed training. Highly appropriate career path. Very Good Very good track record at every step of research pathway. Progressing at very good pace and good depiction of research career goals and relevance of the proposed training. Logical career path. Good Good track record at every step of research pathway. Progressing at good pace and reasonable depiction of research career goals and relevance of the proposed training. Suitable career path. Score Adequate Less Than Adequate Not Acceptable MSFHR 2015 Research Trainee Competition: Evaluation Criteria for Reviewers Page 9

10 Evaluating the Reference Letters Weighting 10% Assessment Criteria Do the reference letters provide evidence of the applicant s personal characteristics, academic strengths and skills that correlate with research career achievement? Are the references provided from diverse sources (e.g. not all from the same department)? Study the referee s assessment, recognizing that positive comments are common while negative ones are not. Look for comments where the referee perceives the applicant as an investigative type, that is, someone whose thinking is critical, independent and original. Has the referee perceived the applicant to be energetic and highly focused? If the applicant has had an opportunity to conduct research, does the referee comment on creativity in setting research goals, designing experiments, interpreting findings and presenting results in writing. Additional criteria for KT science projects only: o Has the referee perceived the applicant to be able to work with diverse user-groups? Assessment Descriptor Outstanding References are unanimously strong in their support of the applicant and provide considerably detailed assessments of the applicant s research strengths/ capabilities, intellectual capacity, and personal characteristics. Applicant is exceptionally focused, energetic and creative. An ideal role model for others. Excellent References are very positive and specific details are provided regarding the recognized or potential research expertise, competence and abilities of the applicant. Applicant is very focused, energetic and creative. An excellent potential for future research leadership. Very Good References provide predominantly positive support about the applicant s personal characteristics, academic strengths, and research abilities. Applicant is definitely focused, energetic and creative. Very good potential for success as an independent researcher. Good References provide credible support about the applicant s personal characteristics, academic strengths, and research abilities. The information provided tends to be more general rather than specific. Applicant seems to be focused, energetic and creative. Above-average potential for a productive career in research. Score MSFHR 2015 Research Trainee Competition: Evaluation Criteria for Reviewers Page 10

11 Adequate Less Than Adequate Not Acceptable MSFHR 2015 Research Trainee Competition: Evaluation Criteria for Reviewers Page 11

12 Evaluating Publication Activity Postdoctoral fellows from all research themes including health professionals with a PhD Health professionals without PhD, pursuing degree or research only Weighting 30% 20% Assessment Criteria Has the applicant published or contributed to original research articles, review articles, patents, books or book chapters, abstracts and/or monographs? Opportunities to publish may vary according to research discipline and life events (e.g. parental leave, etc.). Note the number of co-authors and the position of the applicant s name in the authors list (importance of this position can vary across disciplines). Consider the scientific impact factor or the rank in a sub-specialty of the journals that the applicant has been published in; has the applicant published in top-level journals? Does the applicant hold patents and copyrights? What is the applicant s role in publications and their estimated percent contribution to the work? Do the publication dates relate to the applicant s education and training? Consider the quality and appropriateness (not just the quantity) of publications. List of abstracts are to be seen as an indication of conference presentation activities. Assessment Descriptor Outstanding All aspects of the applicant s publications (number, likely impact and breadth) indicate an extraordinarily productive and creative individual in their respective research field/discipline. Excellent Several aspects of the applicant s publications (number, likely impact and breadth) indicate excellent productive and creative individual in their respective research field/discipline. Very Good At least some aspect of the applicant s publications (number, likely impact and breadth) indicates very good productivity in their respective research field/discipline. Good There is evidence of greater than expected involvement in publication activities in their respective research field/discipline. Score Adequate Less Than Adequate Not Acceptable MSFHR 2015 Research Trainee Competition: Evaluation Criteria for Reviewers Page 12