Science’s reproducibility crisis.
For any study to have legitimacy, it must be replicated, yet only half of medical studies celebrated in newspapers hold water under serious follow-up scrutiny — and about two-thirds of the “sexiest” cutting-edge reports, including the discovery of new genes linked to obesity or mental illness, are later “disconfirmed.”
Though erring is a key part of the scientific process, this level of failure slows scientific progress, wastes time and resources and costs taxpayers excesses of $28 billion a year, writes NPR science correspondent Richard Harris/
The single greatest threat to science right now comes from within its own ranks. Last year Nature, the prestigious international science journal, published a study revealing that “More than 70% of researchers have tried and failed to reproduce another scientist’s experiments, and more than half have failed to reproduce their own experiments.”
The inability to confirm research that was published in highly respected, peer-reviewed journals suggests something is very wrong with how science is being done.
The crisis afflicts even science’s most revered ‘facts,’ as cancer researchers C. G. Begley and Lee Ellis discovered. Over an entire decade they put fifty-three published “landmark” studies to the test; they succeeded in replicating only six – that’s an 11% success rate.
A major culprit, they discovered, is that many researchers cherry-picked the results of their experiments – subconsciously or intentionally – to give the appearance of success, thereby increasing their chances of being published.
“They presented specific experiments that supported their underlying hypothesis, but that were not reflective of the entire data set,” report Begley and Ellis, adding this shocking truth: “There are no guidelines that require all data sets to be reported in a paper; often, original data are removed during the peer review and publication process.”
Another apparent culprit is that – and it’s going to surprise most of you – too many scientists are actually never taught the scientific method. As graduate students, they take oodles of courses in their chosen specialty; but their thesis advisors never sit them down and indoctrinate them on best practices. Consequently, remarks University of Wisconsin-Madison biologist Judith Kimble: “They will go off and make it worse.”
This observation seems borne out by the Nature study, whose respondents said the three top weaknesses behind science’s reproducibility crisis are: 1) selective reporting, 2) pressure to publish, and 3) low statistical power or poor analysis. In other words, scientists need to improve on practicing what they preach, which is: 1) a respect for facts – all of them, not just the ones they like, 2) integrity, and 3) a sound scientific method.
The attendees of the so-called March for Science made a lot of noise about wanting more money and respect from the public and government – what group wouldn’t want that? But nary a whisper was heard from them or the media about science’s urgent reproducibility crisis. Leaving unspoken this elephant-sized question: If we aren’t able to trust the published results of science, then what right does it have to demand more money and respect, before making noticeable strides toward better reproducibility?