What crisis ?

I've mentioned that I'm skeptical of the so-called "replication crisis" before, and this article, ironically, only reinforces that skepticism. The chief editor of the Lancet, after a symposium on reproducibility and reliability, had this to say :

A lot of what is published is incorrect. I m not allowed to say who made this remark because we were asked to observe Chatham House rules. We were also asked not to take photographs of slides. Why the paranoid concern for secrecy and non-attribution? Because this symposium on the reproducibility and reliability of biomedical research touched on one of the most sensitive issues in science today: the idea that something has gone fundamentally wrong with one of our greatest human creations. The case against science is straightforward: much of the scientific literature, perhaps half, may simply be untrue. Afflicted by studies with small sample sizes, tiny effects, invalid exploratory analyses, and flagrant conflicts of interest, together with an obsession for pursuing fashionable trends of dubious importance, science has taken a turn towards darkness. As one participant put it, poor methods get results.

But what exactly is being claimed here ? Either scientists are unknowingly using poor methods/data, or they're doing so in full awareness of what they're doing.

If they don't know what they're doing wrong, then what's the problem ? This is perfectly normal scientific practise, which by its very definition learns from its mistakes. No-one ever told me I had to publish results which were correct - blimey, if that was a requirement, nobody would ever publish anything. It's part and parcel of the whole scientific edifice to challenge existing findings. Research is supposed to be controversial and uncertain. That's the whole point of it.

What should be correct is only that if someone uses my exact method with the exact same conditions on my exact sample, then and only then should they get identical findings (and you can't really even do that in psychology, where conditions can never be fully and exactly reproduced). A paper should stand up to its own scrutiny - the reviewer should be able to say, hand on heart, "I can't find anything obviously wrong with this", but it would be absurd to demand they could say, "these results shall stand for all time and can be extrapolated wildly." The (admittedly few) examples of reproducibility studies I've looked at don't even seem to try and account for this, freely using different samples and/or methods.

And if scientists are ignorant but journal editors are not, why aren't journal editors intervening ? If they really believe the methods being used are known to be flawed, why the hell aren't they doing something about it ? It's not like it's a huge burden to at least say, "this correlation is weak, you should note that somewhere". Or, if the reviewers and editors didn't know about the problems either, then those problems can't have been obvious, and there's no fundamental problem. In that case the whole "crisis" would be just another, routine example of science improving itself.

But this whole thing has been so widely reported that it's not really credible that either authors or reviewers don't know about it either. So what if they're committing the errors knowingly ? Is that even possible ? I can see how the pressure to publish could drive authors to make mistakes and even try for outright fraud, but I cannot for the life of me see what would encourage a reviewer or editor to accept this. And even those authors who do try and deliberately push through dodgy results ought to be only the most desperate, because eventually the results will be discredited by someone else. Half of all authors behaving like this ? Come on, that's silly.

What of the reviewers ? In my experience, most reviewers tend to be overly thorough. Are they massively overworked in the life sciences, such that they're missing rudimentary problems ? I suppose it could be, although reviewing is supposed to be a semi-voluntary activity. It doesn't seem plausible.

The whole thing just doesn't sit right. Neither prospect - that authors are ignorantly making daft mistakes but no-one's telling them about it, or that half of all authors are deceptive little shits - seems at all credible. Is it not, perhaps, infinitely more likely that results aren't reproducible simply because biology is very, very complicated ? Am I missing something here ? If so, please tell me what it is, or else I'll only keep making the same mistakes. Just like the academic authors, apparently.

Science Publication Is Hopelessly Compromised, Say Journal Editors

A lot of what is published is incorrect. Quite an assertion, since it refers to medical progress as a swamp of distortions masquerading as fact, evidence, peer-reviewed science. Who says so? Why, none other than the Editor-in-Chief of The Lancet, among others.