Skip to Content

Navigating the Reproducibility Rapids to a "Learning Research System"

Reproducibility has emerged as a touchstone controversy within the scientific community, the public and the Congress over the past few years. In the wake of recent reports and headlines, science has come under intense scrutiny. But revelations about failure to reproduce experiments, alone, should not shake our trust in the legitimacy of the medical research enterprise. In fact, the attention may be a good thing. As scientists, we understand the complexities of science, but understanding among ourselves will not reassure the public and Congress. If we truly want to engender the public trust and enhance our own learning, let’s welcome this opportunity to address limitations and uncertainties and improve our processes and communication in science rather than overlooking, ignoring, suppressing or railing against the topic.

The term itself, “reproducibility,” can provoke an array of reactions - from immediate confidence (the results were reproducible and so are credible!) - to consideration of the context of the results (differences due to biological variables and applicability of the model, e.g. Were the results obtained in male humans, animals, and cells not generalizable to female species? Was the animal model simply not applicable to humans? Are there discrete differences in the population studied that masked important differences?) - to unfortunate misperceptions (science is fickle. The results must be fraudulent).

Perceptions in the public and in Congress shape policies. To that end, the NIH has just taken a big step toward ensuring reproducibility by increasing the requirements for rigor and transparency in the research that it funds.

At the 2015 GREAT Group and GRAND Professional Development Meeting, prior to the release of the new policy, NIH principal deputy director, Larry Tabak described the NIH’s proposed solutions within the policies to enhance reproducibility: 1) being accountable for adequately describing the methods or materials; 2) noting limitations in the study design; 3) considering all relevant biological variables in the scope of the research (the NIH specifically called out that sex is a biological variable and focusing studies on male animals and cells may obscure the importance of sex on biological processes and responses to interventions); 4) authenticating key biological and chemical resources; and 5) bearing in mind implicit biases in how the results are interpreted or disseminated. Thankfully, from the evidence available, fraud or misrepresentation of findings accounts for only a small percentage of the problems identified, and the scientific community and federal sponsors have effective means to address misconduct.

Navigating the reproducibility rapids goes beyond this policy. There are additional steps the entire research community can take to advance rigor and transparency; and that is by promoting and sharing negative findings as a rule. Beyond enhancing a sense of openness, there may be vital scientific knowledge residing in those negative findings – a legitimate source of “irreproducibility” that could help dissect when and for whom certain interventions may be promising and when and for whom they may not. And, isn’t that a clear signpost for building public trust?

This will take all of us. Sharing data (both negative and positive) would need to be a consideration in merits and promotions, welcomed for publication in respected journals, and established in trusted infrastructures and venues where data sharing is not a prohibitive time and resource burden for the scientists who do the work, or a concern for not protecting privacy in human studies.

In the end, navigating the reproducibility rapids may lead us not only to public confidence and supportive policies, but also to a “learning research system.”