This is a nice, detailed, statistically careful piece.
Two main conclusions can be drawn: (1) experiments reported in high-ranking journals are no more methodologically sound than those published in other journals; and (2) experiments reported in high-ranking journals are often less methodologically sound than those published in other journals.
The prestige, which allows high ranking journals to select from a large pool of submitted manuscripts, does not provide these journals with an advantage in terms of reliability. If anything, it may sometimes become a liability for them, as in the studies where a negative correlation was found. This insight entails that even under the most conservative interpretation of the data, the most prestigious journals, i.e., those who command the largest audience and attention, at best excel at presenting results that appear groundbreaking on the surface. Which of those results will end up actually becoming groundbreaking or transformative, rather than flukes or frauds, is a question largely orthogonal to the journal hierarchy.
Well, a prestigious journal will only accept exciting claims, by definition. And exciting claims are much more common at low reliability levels. This isn't really news, but it's nice to have statistical confirmation of a hitherto largely anecdotal claim.
It is up to the scientific community to decide if the signal-to-noise ratio in these journals is high enough to justify the cost of serial scandals and, in the case of medical journals, loss of life, due to unreliable research.
See also http://astrorhysy.blogspot.cz/2017/03/this-is-not-crisis-youre-looking-for.html
https://news.ycombinator.com/item?id=16427990
https://www.frontiersin.org/articles/10.3389/fnhum.2018.00037/full
I agree. This is one example where competition leads to poor results. You see the same thing in the general media. If a publication doesn't get attention it doesn't sell. If it doesn't sell, the publisher goes bankrupt. Solution: sell the sizzle not the steak. This kind of thing shouldn't happen in science.
ReplyDelete