Reproducibility Problems Run Deep

The reproducibility crisis continues to provide intriguing insights into how to get science back on track.

First flagged by some as a potential problem with peer review or an indictment of glamour journals, further explorations have found that the problems run much deeper than publishing and distribution outlets.

At the recent PSP meeting in Washington, DC, a speaker from the Global Biological Standards Institute (GBSI) explained how 15-36% of cell lines used in biological research are not authenticated. This could be a major contributor, if not the major contributor, to the reproducibility crisis. Other factors he flagged include poor study design, poor data analysis and reporting, and problems with laboratory protocols.

The existence of Eroom's Law (Moore's Law spelled backwards) is especially vexing, and points to fundamental problems that start well before any papers are written or submitted. Eroom's Law points to the approximate halving (between 1950 and 2010) in the number of new drug molecules approved by the FDA per billion dollars of inflation-adjusted R&D investment by the drug industry, despite huge gains in knowledge and brute-force research capacity (e.g., the ability to sequence genes or synthesize chemicals).

In a recent paper from analysts specializing in this area, a set of profound and fundamental problems emanating from biomedical and pharmaceutical research is described:

  • Pursuit of animal models with low predictive value
  • Clinical conditions that aren't described specifically enough yet for targeted therapies to have addressable therapeutic targets, yet which are pursued nonetheless (e.g., Alzheimer's disease)
  • Ignoring "field observations" (i.e., physician reports) of what works, and pursuing reductionist predictive models instead
  • Following management's demand for more R&D throughput rather than ensuring predictive values are sufficient to better ensure success (quantity over quality)
  • Ignoring "domains of validity" for predictive models, and expanding or elaborating upon them inappropriately in a research project
  • Using terminology without rigor, creating confusion or misinterpretations in cross-discipline teams

Journals exist to document and maintain the record of scientific achievements. When these achievements are underwhelming or fraught for whatever reason, the record will reflect this. These and other inquiries into the problem reiterate that the reproducibility crisis is a problem within science, which journals only reflect.

However, as part of the academic and research establishment, journals do have a role in helping to turn things around. More statistical analysis, demanding more explanations of the predictive value of the experiments and the predictive models and their domains of validity can all help. This means spending more time with each paper, and emphasizing quality over quantity.

As Derek Lowe wrote in his "In the Pipeline" blog at Science Translational Medicine:

If you want better, more reproducible papers, you’re going to have fewer of them. Shorter publication lists, fewer journals, and especially fewer lower-tier journals. The number of papers that are generated now cannot be maintained under more reproducible conditions . . .

Or, as the authors wrote in an important Science paper on reproducibility:

Correlational tests suggest that replication was better predicted by the strength of the original evidence than by characteristics of the original and replication teams.

In other words, better evidence is better evidence, and is more likely to be reproducible.

Unfortunately, until the underlying cultural aspects that treasure quantity of publications over quality of publications are fundamentally addressed and changed -- and all the associated economic, financial, reputational, and career issues all players tacitly support -- we will continue to have problems reproducing weak science. Publishers can't solve these problems alone.