Jeffrey W. Elias, PhD UCD SOM Office of Research Grants Facilitation.

Transcription

1 NIH IS CHANGING/ENHANCING THE REVIEW CRITERIA AND FOCUSING ON MISSING OPPORTUNITIES TO EXAMINE SEX DIFFERENCES JOURNALS ARE CHANGING/ENHANCING THE REVIEW CRITERIA Jeffrey W. Elias, PhD UCD SOM Office of Research Grants Facilitation

2 Sex Differences are part of Reproducibility Issues in Scientific Studies Changes in the NIH 2016 submission/review cycle 1) NIH enhances reproducibility standards for application science and review primary issues of bias in design (to include exclusion of sex differences) and bias in reporting data. 2) Focus on enhancing animal research to include sex differences 3) Journal standards for enhancing reproducibility less obvious emphasis on sex differences.

3 Sex Differences are part of Reproducibility Issues in Scientific Studies Sex refers to a set of biological attributes in humans and animals. It is primarily associated with physical and physiological features including sex chromosomes, gene expression, hormone levels and function, and reproductive/sexual anatomy. Sex is usually categorized as female or male but there is variation in the biological attributes that comprise sex and how those attributes are expressed.

4 Sex Differences are part of Reproducibility Issues in Scientific Studies Gender refers to the socially constructed roles, behaviors, expressions and identities of girls, women, boys, men, and gender diverse people. It influences how people perceive themselves and each other, how they act and interact, and the distribution of power and resources in society. Gender is usually conceptualized as a binary (girl/woman and boy/man) yet there is considerable diversity in how individuals and groups understand, experience and express it.

5 Addressing Reproducibility Issues to Include Sex Differences BIAS: From a personal perspective as a researcher, journal reviewer, study section reviewer, and journal editor the most frequent source of bias is a hypothesis: 1) hypotheses often ignore sources of variance that should be known or explored (i.e., lazy hypothesizing). We have been lazy about sex differences. 2) investigators try to fit the data to the hypotheses as proposed and do not follow the results, which is a form of biased reporting.

6 Addressing Reproducibility Issues to Include Sex Differences Funding Opportunity PA , Administrative Supplements for Research on Sex/Gender Differences The Office of Research on Women s Health (ORWH) Most institutes have signed on to PA Due March 4 non-competing administrative supplements 100K Must have 18 months remaining on parent grant

7 Addressing Reproducibility Issues to Include Sex Differences PA , Administrative Supplements for Research on Sex/Gender Differences In 1993, the NIH Revitalization Act required the inclusion of women in NIH-funded clinical research. (from supplement on sex differences) There has not been a corresponding revolution in experimental design and analyses in cell and animal research despite multiple calls to action. Publications often continue to neglect sex-based considerations and analyses in preclinical studies.

8 Addressing Reproducibility Issues to Include Sex Differences PA , Administrative Supplements for Research on Sex/Gender Differences The NIH plans to address the issue of sex and gender inclusion across biomedical research multi-dimensionally through program oversight, review and policy, as well as through collaboration with stakeholders including publishers. This move is essential, potentially very powerful and need not be difficult or costly.

9 Addressing Reproducibility Issues to Include Sex Differences M. Parikh, (2015), Subject selection bias in animal studies, Current Science, Vol. 109, No August. Stroke events are more prevalent in females compared to males, but female animals are used in only 38% of studies. Hypertension is found to be equally present in both sexes, but 65% of scientific reports have males as the study subjects. Male biases were found in 8 out of 10 biological fields.

10 Addressing Reproducibility Issues to Include Sex Differences M. Parikh, (2015), Current Science. 80% of the studies in surgical journals used male animals Women are 1.5 times more susceptible to pain than men and women show higher adverse drug reactions, but still studies use males exclusively, which resulted in withdrawal of 8 out of 10 prescription drugs from the US market as side effects manifested differently in both sexes..

11 Addressing Reproducibility Issues to Include Sex Differences Can this effort be successful? Canadian Institute of Health Research (CIHR) requires its applicants to answer mandatory questions on sex and gender; unlike the NIH, it does not mandate the inclusion of specific populations in research designs, nor does it have corresponding peer review criteria on sex and gender

12 Addressing Reproducibility Issues to Include Sex Differences Can this be successful? In recognition of the important influences of sex and gender on health, in December 2010 CIHR made a change to its grant application forms, requiring that all applicants respond to two questions: Are sex (biological) considerations taken into account in this study? Are gender (socio-cultural) considerations taken into account in this study?

13 Addressing Reproducibility Issues to Include Sex Differences Canadian Institute of Health & Research motivations for change. With ratios of male-only against female-only studies the ratios were: 5.5 in neuroscience, 5.0 in pharmacology and 3.7 in physiology. 75% articles in reputed immunology journals did not indicate the sex of animals used in the study.

14 Figure 2. Applicant by research area, competition and responses to sex and gender items. Johnson J, Sharman Z, Vissandjée B, Stewart DE (2014) Does a Change in Health Research Funding Policy Related to the Integration of Sex and Gender Have an Impact?. PLoS ONE 9(6): e doi: /journal.pone

15 Suggestions for approaching the enhancement of rigor and sex differences mandated by NIH READ: 1) NOT-OD for further consideration of NIH expectations about sex as a biological variable. 2) General FAQs from NIH found at 3)Most recent article by Clayton & Collins for tips on developing sex differences approach. Clayton, J. & Collins F. (2014), NIH to balance sex in cell and animal studies.nature, Vol. 509, 15 MAY 2014,

16 Suggestions for approaching the enhancement of rigor and sex differences mandated by NIH Read: Landis et al (2012) A call for transparent reporting to optimize the predictive value of preclinical research. Nature, Oct 11;490(7419): doi: /nature11556.

17 Suggestions for approaching the enhancement of rigor and sex differences mandated by NIH Pay attention to: By November 25, 2015 application guide instructions will be updated to include the following additional guidance for. Significance Describe the scientific premise for the proposed project, including consideration of the strengths and weaknesses of published research or preliminary data crucial to the support of your application. Approach Describe the experimental design and methods proposed and how they will achieve robust and unbiased results.

18 Suggestions for approaching the enhancement of rigor and sex differences mandated by NIH Pay attention to: Using too subtle an approach to enhanced review issues But I did mention it on page 102! Make reviewers pay attention. Be clear if you are developing hypotheses (e.g., sex differences) or testing hypotheses. Don t be lazy in literature review.

19 Suggestions for approaching the enhancement of rigor and sex differences mandated by NIH Pay attention to: adding sex as a factor in statistical design. There are seven effects involved in a threeway factorial Three main effects (one for each independent variable) Three 2-way interactions One 3-way interaction

20 Suggestions for approaching the enhancement of rigor and sex differences mandated by NIH Pay attention to: adding sex as a factor in a statistical design. There are 15 effects involved in a four-way design Four main effects (one for each independent variable) Six 2-way interactions Four 3-way interactions One 4-way interaction

21 Suggestions for approaching the enhancement of rigor and sex differences mandated by NIH Pay Attention To: Major concern for NIH grants and clinical trials - the focus is on detail of methods. No additional application pages are allotted. Reviewers are refocused before the first round of reviews after January 16, Work in progress for reviewers and applicants.

22 Landis et al (2012) NINDS Nature. Author manuscript; available in PMC 2013 Apr 11. Published in final edited form as: Nature Oct 11; 490(7419): doi: /nature11556 PMCID: PMC NIHMSID: NIHMS A call for transparent reporting to optimize the predictive value of preclinical research

23 Landis Enhancing Animal Research The main workshop recommendation is that at a minimum studies should report on sample-size estimation, whether and how animals were randomized, whether investigators were blind to the treatment, and the handling of data.

24 Landis Enhancing Animal Research Standards Encourage the use of community-based standards (such as nomenclature standards and reporting standards like ARRIVE), where applicable. Replicates Require that investigators report how often each experiment was performed and whether the results were substantiated by repetition under a range of conditions. Sufficient information about sample collection must be provided to distinguish between independent biological data points and technical replicates.

25 Landis Enhancing Animal Research Core set of standards for rigorous reporting of study design (Adapted from Landis et al.) Include these reporting standards in Information for Authors or other public place. Require authors to fill out a checklist, ideally upon submission, to state where the required information is located in the manuscript.

26 Landis Enhancing Animal Research Blinding Require authors to state whether experimenters were blind to group assignment and outcome assessment, at a minimum for all animal experiments. Sample-size estimation Require authors to state whether an appropriate sample size was computed when the study was being designed and include the statistical method of computation. If no power analysis was used, include how the sample size was determined.

27 Landis Enhancing Animal Research Inclusion and exclusion criteria Require authors to clearly state the criteria that were used for exclusion of any data or subjects. Include any similar experimental results that were omitted from the reporting for any reason, especially if the results do not support the main findings of the study. Describe any outcomes or conditions that were measured or used and are not reported in the results section.

28 Landis Enhancing Animal Research Statistics Require that statistics be fully reported in the paper, including the statistical test used, exact value of N, definition of center, dispersion and precision measures (e.g., mean, median, SD, SEM, confidence intervals) Randomization Require authors to state whether the samples were randomized and specify method of randomization, at a minimum for all animal experiments.

29 Reproducibility Journal Standards Journals should have no limit or generous limits on the length of methods sections (including online options), while at the same time encouraging efficient and clear presentation to ensure a thorough examination by reviewers. Journals should use a checklist during editorial processing to ensure the reporting of key methodological and analytical information to reviewers and readers. (A proposed set of key information is listed below).

30 Reproducibility: Proposed Journal Standards Rigorous statistical analysis A section outlining the journal s policies for statistical analysis should be included in the Information for Authors, and the journal should have a mechanism to check the statistical accuracy of submissions.

31 Reproducibility Journal Standards Data and material sharing Stipulate, at the minimum, that all datasets on which the conclusions of the paper rely must be made available upon request (where ethically appropriate) during consideration of the manuscript (by editors and reviewers) and upon reasonable request immediately upon publication. Recommend deposition of datasets in public repositories, where available. Datasets in repositories should be bidirectionally linked to the published article in a way that ensures proper attribution of data production.

32 Reproducibility Journal Standards Encourage presentation of all other data values in machine readable format in the paper or its supplementary information. Require materials sharing after publication. Encourage sharing of software and require at the minimum a statement in the manuscript describing if software is available and how it can be obtained.

33 Reproducibility Journal Standards Consideration of refutations Have a policy stating that if the journal publishes a paper, it assumes responsibility to consider publication of refutations of that paper, according to its usual standards of quality.

34 Reproducibility Journal Standards Consider establishing best practice guidelines for: Image based data (image screening for manipulation, Western blots, for example) Description of biological material with enough information to uniquely identify the reagents (for example unique accession number in repository), in particular for: antibodies: also report source, characteristics, dilutions and how they were validated cell lines: also report source, authentication and mycoplasma contamination status animals: also report source, species, strain, sex, age, husbandry, inbred and strain characteristics of transgenic animals

35 Reproducibility Journal Standards antibodies: also report source, characteristics, dilutions and how they were validated cell lines: also report source, authentication and mycoplasma contamination status animals: also report source, species, strain, sex, age, husbandry, inbred and strain characteristics of transgenic animals