A nicely balanced article that completely fails to jump on the EVERYONE HAS SCREWED UP SO HARD bandwagon. Well, done Guardian, I salute you.
Personally I think we've got a long way to do before we properly understand what "statistically significant" really means. Far too much trust in raw numbers and not enough assessment of context, selection effects and other biases and stuff.
Originally shared by Event Horizon
Something rotten in the state of social science ? Even social science is bound by a social context and the implicit psychological and cultural biases of competitive, professional environments. It's an interesting reflection that some academic papers and studies that might be used to inform (among other things) public policy decisions may be as subject to exaggerated significance as are many other contested cultural and professional narratives. Science does not occur in a cultural and economic or value-free vacuum. Questions could be asked of the extent to which cultural, professional and economic pressures to publish papers provides opportunities for false conclusions to be preemptively celebrated as significant. It is an interesting issue.
In total, the team tried to replicate one main finding from each of the 21 social science papers published between 2010 and 2015 in Science or Nature, widely regarded as the two most prestigious scientific journals.
They found evidence to back the original conclusions in 13 of the 21 (62%) studies. But, on average, the sizes of the effects recorded were about 75% as big in the replication studies, despite these using sample sizes that were on average five times as big.
“These results show that ‘statistically significant’ scientific findings need to be interpreted very cautiously until they have been replicated even if published in the most prestigious journals,” said Magnus Johannesson of the Stockholm School of Economics, another of the project leaders.
The latest work revealed scientists were also uncannily accurate at predicting which studies would later succeed or fail to replicate. About 200 scientists were recruited and on average predicted the replication outcomes for 18 out of the 21 papers under scrutiny.
Prof Malcolm Macleod, a neurologist at the University of Edinburgh who has previously investigated reproducibility in biomedical science, said there was a need to prioritise the quality of science as well as the novelty of findings. “We need to wean ourselves off the nectar and the crack cocaine of highly exciting results and work out what we can do to maximise the quality” he said. “That’s becoming much more of a thing now.”
This is clearly what happens when you are willing to let the truth get in the way of a good story...
😆
https://www.theguardian.com/science/2018/aug/27/attempt-to-replicate-major-social-scientific-findings-of-past-decade-fails
I pay for Guardian. Not for subscription, no, for journalism. I chose to donate small sum per month to them. Great work they do, yes. Please support that, too.
ReplyDeleteI don't subscribe or donate to any news outlets. Subscription tends to be unanimously far, far too expensive. The donation model is much better, but it's inconvenient. What I want is a service where I can subscribe to multiple sources at once, at a modest level. Something where I could get a useful but limited number of articles per month (the standard free access seems to be 2-5, which is way too small, 10 would be about right) at a much reduced fee compared to full access.
ReplyDeleteBiomedical journals went through a similar crisis ~20 years ago*, and it's gotten a lot better since. Not perfect, but better, and it continues to move that way.
ReplyDelete* about the time my career started. Coincidence? ;-)
Rhys Taylor What's inconvenient about allowing them to check out 5 or 10 bucks a month from your paypal? Yes, I don't like paypal either, but it's the most convenient. I do similar with webradios. Works fine, is better than any "legal" broadcast stuff.
ReplyDelete