The NIH Peer Review System

Transcription

1 The NIH Peer Review System Dolores J. Lamb Director, Center for Reproductive Medicine, Lester and Sue Smith Chair in Urologic Research Vice-Chairman for Research and Professor in the Scott Department of Urology, Molecular and Cellular Biology, Baylor College of Medicine, Houston, TX

2 NIH PEER Review The NIH dual peer review system is mandated by statute in accordance with section 492 of the Public Health Service Act and federal regulations governing "Scientific Peer Review of Research Grant Applications and Research and Development Contract Projects" (42 CFR Part 52h). NIH policy is intended to promote a process whereby grant applications submitted to the NIH are evaluated on the basis of a process that is fair, equitable, timely, and free of bias.

3 First Level of Review Scientific Review Officer Examine each application and check for completeness Document and manage COIs Recruit qualified reviewers Assign applications to reviewers Attend and oversee administrative and regulatory aspects Prepare summary statements for all applications reviewed

4 Scientific Review Group Members Chair-moderator & Peer reviewer Reviewers Declare COIs Prepare a written critique Make recommendations of scientific and technical merit (numerical scores) Make recommendations on human subjects, animal welfare, other areas applicable to the review Make recommendataions regarding the budget

5 Other Attendees NIH Staff with need-to-know or pertinent responsbilities Other federal staff members with advance approval Programmatic or grants management staff

6 Peer Review Criteria Varies According to the Type of Grant Reviewed- But in General Overall Impact (1 Exceptional- 9 poor) Scored Criteria Significance Investigator Innovation Approach Environment

7 Significance Important problem? How will scientific knowledge, technical capability, and/or clinical practice be improved? How will successful completion of the aims change the concepts, methods, technologies, treatments, services, or preventative interventions that drive this field?

8 Investigator Well-suited to project? If new investigator, do they have appropriate experience and training? If established investigator, is there an ongoing record of achievement? If multi-pd/pk, do investigators have complementary expertise? Leadership approach and organizational structure appropriate for the project?

9 Innovation Does the proposal challenge existing paradigms? Use new novel theoretical concepts, approaches, or interventions? Do the approaches bring concepts or methods from one field to another? Are there new methodologies or interventions proposed?

10 Approach Are the Specific Aims, overall strategy, methodology appropriate to accomplish the aims? Did the applicant consider problems, alternative approaches, expected results and benchmarks for success? What is feasibility of successful outcome and if risky, how will PI manage this risk?

11 Environment Does the scientific environment offer to enhance the probably of successful completion? Cores, collaborators, equipment, insitutional commitment Will the project benefit from environment, subject populations, collaborative arrangements?

12 A Review Template is Used SCORED REVIEW CRITERIA Reviewers will consider each of the five review criteria below in the determination of scientific and technical merit, and give a separate score for each. 1. Significance Strengths Weaknesses 2. Investigator(s) Strengths Weaknesses 3. Innovation Strengths Weaknesses

13 Peer Review of Training Grants The Applicant Productive, well-trained, awards, honors, publications? The Mentor(s) Experienced (i.e. not a new Asst. Professor), productive, strong scientist, well-funded, # previous trainees and their track record? The Environment Scientific cores, seminars, workshops, education opportunities The Training Potential What will the trainee learn and how will this impact their career development? The Scientific Merit The significance, innovation, feasibility, approach,

14 Additional Review Criteria Protections for Human Subjects Inclusion of Women, Minorities, and Children Vertebrate Animals Biohazards Resubmission Renewal Revision

15 Additional Review Considerations Applications from Foreign Organizations Select Agent Resource Sharing Plans Budget and Period Support

16 Important Tips to Remember: Reviewers May have 8-10 proposals to critique This is not their Day-Job May not know ANYTHING about your specific research area Make the proposal EASY for them to review Well-written Simple and clear Flow-charts and diagrams can provide a great deal of information Spend a great deal of time on developing the science AND how the proposal looks and reads

17 Important Tips to Remember Did you state your hypothesis? Did you describe your experimental design (not method) including positive and negative controls, data analysis and interpretation? Did you appropriately cite the literature? Did you label Figure legends and perform statistics on data shown? Are fonts readible? Too small?

18 Who Are Your Reviewers? Standing Committee Members Broad diversity of expertise and experience Sometimes surprising mix of individuals Roster is publically available Ad Hoc Reviewers Bringing needed expertise to the panel Final Roster is posted 1 month prior to review

19 Some Reviewers May Provide Expertise in One Discrete Area Key to a Proposal Statistical Review Bioinformatics Genetics Molecular Biology Urology And so on. Some reviewers may only mail in or phone in their critique on a single assigned proposal if unique expertise is needed

20 Scoring Prior to the meeting each reviewer and discussant posts their overall and review criteria scores Reviewers read the critiques and scores on assigned grants prior to meeting Reviewers may amend scores or critiques prior to the meeting based upon the other critiques

21 Scoring Prior to the meeting the Preliminary Score Matrix is available allowing reviewers to see the spread of scores and incongruous scoring The SRA circulates a proposed lower half list of the 50% of the proposals scoring lowest At the meeting, this list is reviewed, a reviewer may always request that a grant be discussed

22 Reviewer Guide to Scoring Impact High Moderate Low Impact/Priori ty Score Descriptor Strengths/Weaknesses 1 Exceptional Exceptionally strong, no weaknesses 2 Outstanding Extremely strong, negligible weaknesses 3 Excellent Very strong, minor weaknesses 4 Very Good Strong but numerous minor weaknesses 5 Good Strong but at least one moderate weakness 6 Satisfactory Some strengths but some moderate weaknesses 7 Fair Some strengths but at least one major weakness 8 Marginal A few strengths and a few major weaknesses 9 Poor Very few strengths, numerous major weaknesses

23 Frequently, the longer the discussion the lower the score At the Review Meeting The applications are discussed in descending order (highest scoring first to lower scoring) It is amazing what panel participants The primary, secondary reviewers and discussant know state and their preliminary the topics scores of this discussion can The be primary critical reviewer to summarizes the overall the project scoring. and This strengths discussion and weaknesses can sway for each the criterion ultimate The secondary reviewer discussant do not scoring repeat similar from critiques an outstanding but focus on their score view of to strengths and a very weaknesses poor that score! differ Any panel member can then join the discussion

24 The Scoring Each Reviewer and Discussant provide their overall impact score for the application This score may differ radically from their preliminary score based upon the discussion that preceded scoring Those voting out of the range of scores must state that they will vote out of range and why their scoring differs

25 Scoring Consensus is not required Final scores are averaged from all panel reviewers and multiplied by is a perfect score Unscored proposals are usually ~60-90 range

26 Scoring Reviewers may see totally different strengths and weaknesses and score appropriately Both can be correct but their overall views simply differ Reviewers may have similar strengths and weaknesses but may score differently Weighting criteria or strengths differently Unless there is complete scoring consensus, it is impossible for reviewers to know a final score

27 Proposals are not compared or ranked when scored- only scientific merit is considered- not relative scoring Scoring is a difficult job and Reviewers may not score well No spread of scores making it impossible to distinguish grant merit at second level of review (council) and by program staff Reviewers may refuse to budge on their preliminary score when other reviewers present critiques that suggest their scoring was off target

28 Summary Statement The Score and its percentile rank (if scored) A percentile is the approximate percentage of applications that received a better overall impact/priority score from that section during the past year. For applications reviewed in ad hoc study sections, a different base may be used to calculate percentiles. All percentiles are reported as whole numbers Only a subset of all applications receive percentiles. The types of applications that are percentiled vary across different NIH Institutes and Centers. The summary statement will identify the base that was used to determine the percentile.

29 Texas Medical Center, Houston