Results in prestigious journals are exciting but not especially reliable

This is a nice, detailed, statistically careful piece.

Two main conclusions can be drawn: (1) experiments reported in high-ranking journals are no more methodologically sound than those published in other journals; and (2) experiments reported in high-ranking journals are often less methodologically sound than those published in other journals.

The prestige, which allows high ranking journals to select from a large pool of submitted manuscripts, does not provide these journals with an advantage in terms of reliability. If anything, it may sometimes become a liability for them, as in the studies where a negative correlation was found. This insight entails that even under the most conservative interpretation of the data, the most prestigious journals, i.e., those who command the largest audience and attention, at best excel at presenting results that appear groundbreaking on the surface. Which of those results will end up actually becoming groundbreaking or transformative, rather than flukes or frauds, is a question largely orthogonal to the journal hierarchy.

Well, a prestigious journal will only accept exciting claims, by definition. And exciting claims are much more common at low reliability levels. This isn't really news, but it's nice to have statistical confirmation of a hitherto largely anecdotal claim.

It is up to the scientific community to decide if the signal-to-noise ratio in these journals is high enough to justify the cost of serial scandals and, in the case of medical journals, loss of life, due to unreliable research.

See also http://astrorhysy.blogspot.cz/2017/03/this-is-not-crisis-youre-looking-for.html

https://news.ycombinator.com/item?id=16427990
https://www.frontiersin.org/articles/10.3389/fnhum.2018.00037/full