Case study of the replication crisis : science is just frickin' hard

This is about the so-called replication crisis in psychology. In this case study, it appears that it's nothing to do with poor analysis techniques at all. It's simply because people are really frickin' complex. Cultural influence and self-belief matters enormously and finding genuine universal properties is hard. Maybe some studies really were badly done, but this one looks like a case of things working exactly the way they're supposed to : a basic claim is made, then further findings start to reveal the nuances of the situation.

As anyone who has ever tried a diet knows, exerting willpower can be exhausting. After a whole day spent carefully avoiding the snack machine and attempting to take mindful joy in plain baked chicken and celery sticks, the siren call of cookies after dinner may be just too much to bear. This idea — that exercising self-control gets harder the more you have to do it — is called ego depletion, and it’s one of the most well-known concepts in social psychology. There are popular books on it. Most of us have probably have personal experience with it.

But what if a huge study of thousands of people found no evidence for ego depletion? What if some cultures actually show reverse ego depletion — where exerting willpower actually makes exerting more willpower easier? What if I told you that ego depletion does exist — but only if you believe it does?

“In psychology, nothing happens all the time. We find stuff that happens sometimes. That’s about as well as we can do.” People have bad days and good ones, sleepless nights and restful nights, good and bad childhoods. “I think the scientific question should be, ‘what are the conditions under which [ego depletion] does and does not happen?’” Baumeister says.

This idea — that sometimes it happens, sometimes it doesn’t — isn’t particularly satisfying. Most of us were taught that, in science, a scientist forms a hypothesis, tests it and then throws it out if it doesn’t work. That’s what the philosopher of science Karl Popper thought, says Janet Stemwedel, herself a philosopher of science at San Jose State University in California. In this view, scientists go out every day and “throw hypotheses in the deep end of the pool to see if they can swim.” By Popper’s standards, if ego depletion fails to replicate, it’s a failed hypothesis. It deserves to drown.

I think everyone knows that abject falsification is an unusual extreme. Much depends on how specific the claim being made is. If it's that "ego depletion is a universal phenomenon and a fundamental feature of the human brain", then that claim is clearly BS and deserves to be shot down (certain issues with the measurement techniques notwithstanding). But if it's the more interesting and vague claim that "ego depletion is a thing that happens sometimes", then saying that it doesn't always happen so the theory is wrong is itself BS : it would be neglecting a very interesting finding about how strong our subjective beliefs influence us. It's right to make claims as specific as possible so that they can be testable, but there's a huge difference between examining the fundamental mechanism proposed and the conditions under which it operates. This happens in physics as well as psychology.

https://astrorhysy.blogspot.com/2017/05/i-told-you-he-was-tricksy.html
https://www.sciencenews.org/blog/scicurious/willpower-ego-depletion-social-psychology-replication