The ability to reproduce experimental results reported in academic and scientific publications sits at the heart of the scientific method. Put simply, if experimental results cannot be reproduced by third-parties, for any reason, then the scientific merit of these results is fundamentally questionable. In the past two decades, there has been a growing awareness of the fact that many reported academic and scientific studies are difficult or even impossible to reproduce. This puts into question the veracity and validity of these studies and inevitably leads to an erosion of public trust in academia and science. The coining of the phrase "Reproducibility Crisis" or "Replicability Crisis" in the past decade was meant to raise awareness of the gravity of the situation so that proper community-wide interventions can take shape to counter this crisis.
As an educator and researcher in computer science and electronics engineering, I have seen first-hand how the reproducibility crisis has grown over the past two decades. In this blog, I will draw upon my own experiences and observations to attempt and explain the reproducibility crisis phenomenon and its causes. I will then present examples of this crisis in different disciplines and its consequences. After that, I will make suggestions to mitigate the impact of this crisis before drawing some conclusions.
In a revealing survey of more than 1,500 scientists conducted by Nature in 2016 [1], 70% of researchers surveyed said they have tried and failed to reproduce other groups' experimental results. Asked about reproducing their own experimental results, more than half said they could not! It is this failure to reproduce so many research results published in the literature that has given rise to what is commonly known these days as the "reproducibility crisis". This is not just an academic concern however, for much of our system of inventions and enterprise relies fundamentally on reproducible research results. Failure to address this problem would question whole systems of academic and scientific research, innovation and enterprise, and ultimately the wider economy.
The reasons behind the failure to reproduce so much of publication results are diverse, and include:
The reproducibility crisis touches a wide range of disciplines in engineering & physical sciences, biological & medical sciences, and social sciences.
Biological & medical sciences are perhaps the academic and research discipline whereby the impact of the reproducibility crisis is mostly felt as it can have a direct incidence on the health of ordinary citizens. For instance, take up of the MMR vaccine in the UK was severely affected after the publication of a controversial study in 1998, which suggested a link between the MMR vaccine and autism, even though the paper reported a small case series with no controls [3]. Subsequent studies refuted this finding, but the public scare the original publication created led to noticeable Measles outbreaks.
In social sciences, Economics is often cited as the prime example. In it, the debate continues to rage around the vast range of economic analyses and forecasts despite an often-common pool of data. For instance, a 2016 study in the journal Science found that one third of 18 experimental studies from two top-tier economics journals failed to be reproduced [4]. When we factor in the importance of economic forecasts in political life and decision making, one could easily see the damage that the reproducibility crisis is doing.
Closer to my own area of research interest, engineering & physical sciences are not immune to the reproducibility crisis. Indeed, selective reporting is still commonplace especially in engineering where there is no strong culture of experimental design reporting as in biological sciences. For instance, it is still not uncommon for benchmarking studies in computer engineering to report on performance e.g. speed, with no reference to trade-offs such as circuit area/code size or power/energy consumption. It is also not uncommon for publications to report on synthetic benchmarks which "artificially" show a particular hardware or software solution in a good light at the expense of competitors. Worse, practical concerns such as cost, sourcing, robustness, extendibility and maintainability are still routinely omitted.
The reproducibility crisis could and should be addressed by going back to the basics of the scientific method. Reproducibility must be a cardinal precondition for scientific publishing. Below are practical ways to achieve that:
None of the above is particularly challenging to implement, the biggest hurdle is ultimately cultural. The academic and scientific community has to develop a collective consciousness around the scientific method and the assessment of research. In particular, we need to align the interests of individual researchers with the collective interest of society. For instance, as long as we continue to reward flawed metrics such as number of publications or even journal impact factors, we will perpetuate the perverse incentives of the "Publish or Perish" culture.
The reproducibility crisis is a major threat to public trust in science and research. A combination of a fall in standards of practice exacerbated by a "Publish or Perish" culture, and a lack of adequate frameworks for the assessment of research quality means that this problem is set to persist if no decisive actions are taken. Essentially, this is a classical collective action problem and the future will ultimately be shaped by any of the following developments (or combinations thereof):
Clearly, self-organization is the most desirable outcome. However, that will only become a reality if the underlying principles and actions are collectively adopted by a large number of researchers and scientists. I posit that a larger weight of responsibility lies on the shoulders of established academics and researchers. If that does not materialize, however, then regulation and/or privatization might well be the only ways forward.
You can keep up with the latest from Arm Education and contribute to the wider debate in the comments sections below.
Arm Education
[1] "Is There a Reproducibility Crisis in Science?", Nature Video, May 28, 2016, https://www.scientificamerican.com/video/is-there-a-reproducibility-crisis-in-science/
[2] Open Research and Publishing: Reflections, Arm Education Media. Jan. 2020.
[3] https://www.bmj.com/content/342/bmj.c7452
[4] "Evaluating replicability of laboratory experiments in economics", C. F. Camerer et al., Science, 25 Mar 2016, Vol. 351, Issue 6280, pp. 1433-1436
[5] https://sfdora.org/read/
[6] http://www.leidenmanifesto.org/
[7] https://www.coalition-s.org