Difficulties replicating scientific studies

Maggy Wassilieff posted: Major overhaul of science and science publishing long overdue.

According to a recent survey published in the journal Nature, more than 70% of researchers have tried and failed to reproduce another scientist’s experiments.

Linking to BBC: Most scientists ‘can’t replicate studies by their peers’

Science is facing a “reproducibility crisis” where more than two-thirds of researchers have tried and failed to reproduce another scientist’s experiments, research suggests.

Has scientific study become too complex to easily replicate? Or are too many studies poorly done?

From his lab at the University of Virginia’s Centre for Open Science, immunologist Dr Tim Errington runs The Reproducibility Project, which attempted to repeat the findings reported in five landmark cancer studies.

“The idea here is to take a bunch of experiments and to try and do the exact same thing to see if we can get the same results.”

You could be forgiven for thinking that should be easy. Experiments are supposed to be replicable.

The authors should have done it themselves before publication, and all you have to do is read the methods section in the paper and follow the instructions.

Sadly nothing, it seems, could be further from the truth.

After meticulous research involving painstaking attention to detail over several years (the project was launched in 2011), the team was able to confirm only two of the original studies’ findings.

Two more proved inconclusive and in the fifth, the team completely failed to replicate the result.

“It’s worrying because replication is supposed to be a hallmark of scientific integrity,” says Dr Errington.

A lot of science must be ok, the world continues to advance through many areas of science. For example new drugs do seem to be effective in part at least.

But if scientific study can’t meet a fundamental requirement of sound research – being able to be replicated – then there should be major concerns.

Concern over the reliability of the results published in scientific literature has been growing for some time.

According to a survey published in the journal Nature last summer, more than 70% of researchers have tried and failed to reproduce another scientist’s experiments.

Is that because of a lack of understanding of how to do the experiments? Or poor initial research?

The reproducibility difficulties are not about fraud, according to Dame Ottoline Leyser, director of the Sainsbury Laboratory at the University of Cambridge.

That would be relatively easy to stamp out. Instead, she says: “It’s about a culture that promotes impact over substance, flashy findings over the dull, confirmatory work that most of science is about.”

Similar in a way to journalists and media tending more towards glitz and click bait and doing less investigative research on things that really matter.

She says it’s about the funding bodies that want to secure the biggest bang for their bucks, the peer review journals that vie to publish the most exciting breakthroughs, the institutes and universities that measure success in grants won and papers published and the ambition of the researchers themselves.

“Everyone has to take a share of the blame,” she argues. “The way the system is set up encourages less than optimal outcomes.”

Hence, presumably, Maggy suggesting “Major overhaul of science and science publishing long overdue.”

“Replication is something scientists should be thinking about before they write the paper,” says Ritu Dhand, the editorial director at Nature.

“It is a big problem, but it’s something the journals can’t tackle on their own. It’s going to take a multi-pronged approach involving funders, the institutes, the journals and the researchers.”

But we need to be bolder, according to the Edinburgh neuroscientist Prof Malcolm Macleod.

“The issue of replication goes to the heart of the scientific process.”

Writing in the latest edition of Nature, he outlines a new approach to animal studies that calls for independent, statistically rigorous confirmation of a paper’s central hypothesis before publication.

“Without efforts to reproduce the findings of others, we don’t know if the facts out there actually represent what’s happening in biology or not.”

Without knowing whether the published scientific literature is built on solid foundations or sand, he argues, we’re wasting both time and money.

“It could be that we would be much further forward in terms of developing new cures and treatments. It’s a regrettable situation, but I’m afraid that’s the situation we find ourselves in.”

So will anything change?

Not only is suspect research a problem – if there are a lot of questions about research reliability then it allows doubts to be raised even over good research.


  1. Alan Wilkinson

     /  February 26, 2017

    Sad but true, Maggy. I practised science in a time when the early German chemists’ publications were regarded as the Edmonds Cookbook of chemical synthesis and unquestionably honest, accurate and reproducible.

    Now we have fuzzy disciplines employing poor statistical analyses on highly dubious datasets with selective publication of sensationalist conclusions.

    However, as far as reproducibility is concerned there seems to be no difficulty in climate science. Every publication confidently asserts that its investigation is compatible with human-induced climate change – even though almost all of them bear no evidence whatever supporting human-induction or predictable climate change.

    – re-posted from OF thread.

  2. Yes Alan. In particular the climate debate has a real problem in establishing the truth of the link between effect and cause. The old conundrum about the truth or fiction of a butterfly flapping its wings in Brazil and causing a tropical storm in the North Pacific. Then the question of carbon dioxide in the atmosphere :
    The global average concentration of CO2 in Earth’s atmosphere is currently about 0.04%, or 400 parts per million by volume (ppm). There is an annual fluctuation of about 3–9 ppm which is negatively correlated with the Northern Hemisphere’s growing season. To what extent does the carbon cycle leach CO2 out of the atmosphere through the process of photosynthesis to create sugars and oxygen, so more CO2 is a plus as long as the trees are left to grow, or cropping areas increased.
    Then we have the consequences of volcanic eruptions and natural dispersal of methane from previous plant matter being converted to fossil fuels.
    Next we have the Sun and the changing rate of cosmic energy dispersal in the troposphere and the effects of the Sun and the Earths rotation and the circulation of the atmosphere and creation of the climate and seasons, plus the Moon and the Tides.
    So the temperature goes up as measured on a specific day. What is the cause. I know the effect, I will feel hotter. But can we blame humankind for the climate change?