Any self-respecting statistician would say that if you tracked the number of times you thought of any friend, and the number of times you had that friend immediately ring you, you’d find the link to be statistically insignificant. But it is not necessarily irrational to attribute grander significance to this occurrence. To those who believe in meaningful coincidences, statistical insignificance does not undermine an event’s causality or importance. To them, just because something could happen doesn’t mean it wasn’t also fated to happen.
Humean logic, in a way : correlation not being equivalent to causation because you can't ever know the true cause. I think it's extremely useful to bear in mind that the cause of something might not be the cause you think it is, but not to abandon the notion of causation completely. I don't believe you can have a scientific world view without accepting physical causality - if you do that you can make up whatever answer you like.
Statistics has limited but important explanatory power. If processes are truly random, you can use pure statistical analysis to explain why coincidences (or indeed any events at all) happen. If they're not at all random you can chuck out the statistics textbook and use pure physics. Such extremes are rare indeed, and usually you need both to explain the physical reasons why some things happen as frequently as they do. This doesn't mean you have to pay any attention to Mystical Woo, but there are usually factors that cause things to happen more frequently than random chance would allow - and these aren't always obvious. If you discount "physics" (using that term very loosely) you could end up with absurdly low probabilities for events that happen quite frequently.
Even if every possible coincidence could be scientifically explained, we shouldn’t necessarily discount its importance. You can watch a movie or read a novel, and be at once aware of its nonreality while also being moved by it. Must these ideas therefore be incompatible? Indeed, might the continued belief in meaningful coincidences even be rational and necessary to our experience of existing in the world? And, is a belief in meaningful coincidences something vital to our survival as humans?
We ascribe exceptional meaning to what we perceive as exceptionally low-probability events, but they’re often not as low-probability as we think. And, even if they are unlikely, the most unlikely events are – with 7 billion people on Earth – actually relatively common, thanks to the so-called law of truly large numbers, the statistical adage of Frederick Mosteller and Persi Diaconis, in which a big sample size will eventually lead to essentially any result... Plus, we’re culturally trained to see meaning in intrinsically meaningless events. Even Plutarch understood this. Writing in ‘The Life of Sertorius’, a volume in his Parallel Lives series (1st century CE), he noted: ‘It is no great wonder if in the long process of time, while fortune takes her course hither and thither, numerous coincidences should spontaneously occur.’
Coincidences, write Yitta Halberstam and Judith Leventhal in their bestselling book Small Miracles: Extraordinary Coincidences From Everyday Life (1997), show ‘the rich promise of a bounteous universe and the splendour lying dormant within your soul. Coincidences are everywhere and can happen any time. When your soul is ready, they will come. All that is required is that you open your heart.’ This is obviously just a bit of feelgood hocus pocus. But it’s not actively harmful either, other than in taking advantage of people’s willingness to pay to hear what they want to hear.
Here I disagree. I think that's an extremely harmful belief that says, instead of being subject to the whims of chance, you're responsible for your own (mis)fortuntes. And that's just not always true, and assigns blame where none should be given. Sometimes you're just unlucky.
Scientists, therefore, need to help ‘people to make decisions on the basis of concrete evidence, not half-baked pseudoscience,’ says Hand, the statistician. ‘Scientists have a public duty to help to disperse the mists of confusion.’
Beneath the statistical incorrectness... a belief in meaningful coincidence is, from an existential perspective, surprisingly rational. If your father were to choke to death across the country at the same time that you felt a phantom choking, you might know, intellectually, that there was no mysterious, invisible connection at play. But, if you did let your mind wander to that possibility, it would allow you a new way to grieve your father’s death, giving you a sense of intimacy or a fatalist understanding of events... behind such a statement is the sometimes-necessary need to fabricate meaning.
https://aeon.co/essays/just-how-meaningful-is-coincidence-beyond-the-statistics
Yesterday, the g/f and I went shopping. On a lark, we took a short detour to a nearby park to see the recent improvements. To get to the park ( as with practically everything else in American life ) required a drive through a newly built cluster of shops and chain hotels.
ReplyDeleteAs we drove past the Staybridge Suites, I remembered a long consulting stint in Mt. Laurel New Jersey, where I'd stayed in a Staybridge Suites hotel. These hotels are all cut from the same bolt of cloth, the back courtyard was precisely the same.
We checked out the park and began to drive back home. My phone rang, it was a headhunter seeing if I was available for another consulting engagement. From Mt. Laurel, New Jersey. Said so right on my phone.
Rather than dismissing the unscientific - and frankly silly - fallacy of these coincidences, I look at them and see the human mind trying to connect the current evidence to its own internal framework of the world, much of which requires bridging enormous gaps for which reason is insufficient.
Cody Delistraty says ‘overfitting’ occurs when you fit your belief model to the noise rather than the signal - and I won't argue with this definition. But to my way of thinking, "Overfitting" is a profoundly different thing:
I utilise neural nets for troublesome little problems which aren't amenable to straightforward rules-based solutions, especially vision systems doing QC. I train 'em by building a monitor process which watches a human doing the job, then feeding the verdict to the neural net. Eventually the neural net can do as good a job as the operator.
But there's a problem: neural nets can be overtrained. Like human beings, they can become dogmatic. Overfitted. Once a neural net has reached that state, the trainer has to relax it somewhat, or retrain it outright on new training evidence. The training data can lead the neural net into a brittle state.
The only way I know to overcome irrational decision making is two semesters of statistics. Nothing else will do. But I'd also advise keeping a sense of humour: the absurd will either make you laugh or break your heart or reduce you to a credulous cargo cultist, with a bone in your nose, sitting at the controls of some wrecked bomber in the jungle, holding the yoke, making VROOOOM VROOOOM noises.