Rigor and transparency in NIH grant proposals

Rigor and transparency are relatively new NIH Reviewer criteria specifications, and important parts of your NIH grant proposal. Here is a summary of how it gets evaluated.

Scientific Premise (in Significance)

Reviewers must consider the quality of the supporting research that the proposal draws upon. Reviewers must evaluate the scientific foundation for the proposed work, and in particular the soundness and rigor of the preliminary studies and/or prior research upon which the proposed study is based. This is scored as a part of the significance evaluation, and does influence overall score.

Make it easy for reviewers to assess scientific premise by including in your significance section a paragraph headed "Scientific Premise". Discuss the pros and cons of the supporting research. Treat this as a mini-systematic review where the quality of the supporting research is critically evaluated (supporting studies' design, sample size and statistical power, sampling, potential for confounding or bias, critical evaluation of hypotheses).

Scientific Rigor (in Approach)

Reviewers are asked to comment on, and incorporate into their scoring of Approach, the methods and study design features and how prone they are to bias and make use of rigorous methodology. Scientific rigor evaluations influence the overall score.

Make it easy for reviewers to evaluate the scientific rigor of your proposal by including a paragraph, in the approach section, headed "Scientific Rigor". In this section, discuss decisions made and rule out potential alternatives that characterize your study design. This might also be a place to describe anticipated problems and strategies to overcome them.

Relevant Biological Variables such as sex

Both males and females (regardless of species) should be included in all research involving living animals (including humans), unless specifically excluded for justifiable scientific reasons. The NIH guidelines for reviewers (see flowchart) specify that, unless sex-specific effects are hypothesized, minimally it is necessary to state and plan to report outcomes separately by gender, but at the same time it is acceptable to test study hypotheses in both samples simultaneously. In my experience reviewers typically do not know this aspect of the guideline. Therefore it is up to the applicant to remind the reviewer and justify the approach.

Authentication of materials

See here. Does not influence score. Special forms required. Key biological or chemical resources must be authenticated. Such materials include, among other things, cell lines, specialty chemicals, antibodies, and other biologics.

Data sharing

NIH guidelines requires "all investigator-initiated applications with direct costs greater than $500,000 in any single year will be expected to address data sharing in their application" (NOT-OD-03-032). In my experience, many reviewers do not know this and will ding any application that does not address data sharing regardless of the funding level. Study section members may or may not call a reviewer on this critique during discussion depending on mood, fatigue, etc. So what is a proposal writer to do? I'm generally in favor of sharing data, so why not make a plan to share the data? A different strategy would be to remind reviewers what is contained in NOT-OD-03-032 in this section. You could do both, and in that way contribute to the education of reviewers and show them that even though it is not required because you are requesting less than $500k direct costs in all years of funding, you are going to share data anyways.

It should be noted that different institutes and centers have different expectations on data sharing. For example the NIMH "expects that applications involving clinical research with human subjects submitted for funding after May 1, 2015 will plan to submit their data to the NIMH Data Archive" (NOT-MH-15-012). So that is not something that is contingent upon funding level and specific to the NIMH. Be sure to look for your anticipated institute and center for specific data sharing policies. This page from the NLM might be a good place to start looking.

References

NIH (2003). Final NIH statement on sharing research data. NOT-OD-03-032.

NIH (2016) Reviewer guidance on rigor and transparency. (PDF)

NIH (2016) Rigor and Reproducibility (Web Page) find at https://grants.nih.gov/reproducibility/index.htm

NIH. (2015). Consideration of sex as a biological variable in NIH-funded research (companion reference). Retrieved from http://orwh.od.nih.gov/sexinscience/overview/pdf/NOT-OD-15-102_Guidance.pdf

NIH. (2015). Consideration of Sex as a Biological Variable in NIH-funded Research [NOT-OD-15-102]. Retrieved from http://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-102.html

NIH. (2015). Enhancing Reproducibility through Rigor and Transparency [NOT-OD-15-103]. Retrieved from http://grants.nih.gov/grants/guide/notice-files/NOT-OD-15-103.html

NIH. (2016). Implementing Rigor and Transparency in NIH & AHRQ Research Grant Applications (NOT-OD-16-011). Retrieved from https://grants.nih.gov/grants/guide/notice-files/NOT-OD-16-011.html

NIMH (2015). Data Sharing Expectations for Clinical Research Funded by NIMH. https://grants.nih.gov/grants/guide/notice-files/not-mh-15-012.html

NLM (2013, 2018). NIH Data Sharing Policies. https://www.nlm.nih.gov/NIHbmic/nih_data_sharing_policies.html

Video (2016) NIH Policy: Enhancing Reproducibility through Rigor and Transparency. Retreived from https://grants.nih.gov/reproducibility/module_1/presentation_html5.html

Rich Jones

last update: 27 Nov 2018