Calling Lizardshit

A short post to summarise two pieces I found, which I think quite nicely help explain why opinion polls sometimes come up with apparently large numbers of people believing batshit crazy ideas.

Incidentally... "bullshit" is well-recognised as meaning not caring about the truth, "batshit" seems already a standard way of emphasising the absurdly stupid, and "horseshit" appears to to generally used as a synonym of "outright lie". Do we need to add "lizardshit" as a term for "stupid poll making dumb statistical mistakes" ? Maybe.

But first, why lizards ? 

The lizardman constant, says the first piece, is the observation that a small fraction of survey responses are literally bullshit. That is, they are totally wrong for a wide variety of reasons : jokes, boredom, ignorance, insanity, mistakes, lazy, malice, innocent misunderstandings, idiocy. It gives numerous examples of which I will quote only a few here :

• 2% of Americans have “never heard of” institutions such as “the police” or the “U.S. Supreme Court”
• 7% say chocolate milk comes from brown cows and 19% that hamburgers come from pigs
• 6–8% of Americans think they could kill a grizzly bear, lion, elephant, or gorilla with their bare hands)
• 5% of atheists are “absolutely” or “fairly certain” that they believe in God
• 14% of undecided voters said Hillary Clinton might be a demon, but they might vote for her; 2% of Clinton supporters said she was and they would
• 1% of Brits who believe you should be imprisoned for 15 years (or more) if you ever fail to wear a seatbelt in your car

Needless to say, the actual fractions of the populace at large that believe in total hogwash like the above is likely orders of magnitude lower than the surveys suggest. That's the point, not that 5% - which appears to be a reasonable value for the Lizardman Constant - really do believe in lizard people.

Below a certain percentage of responses, for sufficiently rare responses, much or all of responding humans may be lying, lazy, crazy, or maliciously responding and the responses are false. This systematic error seriously undermines attempts to study rare beliefs such as conspiracy theories, and puts bounds on how accurate any single survey can hope to be... The reality is that humans don’t answer questions reliably, accurately, or honestly even close to 100% of the time, and shamelessly fail ‘common sense’ or ‘logic’ or ‘arithmetic’ questions all the time, requiring extensive precautions, careful survey design, and just throwing out a lot of data as garbage.

4-5% isn't so small. So when you get down to even, say, 20% or so, a significant fraction of those responses will be pure bullshit. This does not mean that surveys in general are full of crap, but more that any one survey might well be. If you want a more accurate result you need meta-surveys, asking different people in different places the same thing in different ways, etc.

Note also that "any one survey" is distinct from "every individual survey". They're not the same thing at all. Some individual surveys do get the right answers, and by way of example, I'll point back to that survey of Tory party members, a majority of whom would be willing to sacrifice their own party for the sake of Brexit. Lizardshit ? Methinks not, given that these were the same people who, subsequently, looked at the various leadership candidates squarely in the eye, and decided that the one with less staying power than a lettuce would be the one most suitable to lead the country.

On to the second piece*, which is a case study of a poll claiming that a quarter of Brits believe the coronavirus was a hoax. This is quite obviously nonsense, but the piece does a nice dissection of how to consider this finding critically. After mentioning Lizardman (only 9% said "definitely", which is not far off the constant), it takes a look at what "hoax" could mean to people :

* I already mentioned this before elsewhere, but felt I needed a more permanent record of it.

There’s a whole spectrum of heterodox views on Covid: those who question lockdowns, those who argue the virus was much less deadly than the medical establishment told us, those who believe governments used Covid as an excuse to enact authoritarian policies, all the way to the far more extreme groups of antivaxxers and those who believe the virus never existed in the first place... I bet a huge number of people answering the very vague question in the poll had other – perhaps still technically untrue, but far less outrageous – things in mind. 

And then it reports that a YouGov poll where the question was worded unambiguously gave numbers deep into Lizardman : 3%. But it goes further, asking to consider the implications of other numbers from the same poll :

7 per cent of people said they had already taken part in such a rally. If that were true and representative of the UK adult population (which is about 53 million: the 67 million total population less the 14 million under-18s), it would mean that something like 3.7 million people had already taken direct action and joined a protest against these supposed conspiracies.

What this is telling us is that this poll is seriously flawed in some way. It doesn't tell us exactly what the issue is, but clearly something is wrong.

The pollsters also asked the public whether they’d heard of the conspiracy newspaper the Light – the subject of a BBC article and podcast released this week. A surprising 14 per cent of respondents (7.4 million people, if that’s representative of the UK adult population) said they’d heard of the newspaper... of those who’d heard of it, 40 per cent said they were subscribers - if this was representative, it would mean that more than 2.9 million people in the UK subscribe to the Light – five times as many as subscribe to the Times. 

But of course, it’s not true. By the time you take 40 per cent of 14 per cent of the sample in the poll, you’re down to just 314 people, and all representativeness is out of the window.

Which is a simple, effective tip for considering if a poll is telling you something is meaningful or not. A poll of a thousand people might give you something truly representative on a clear-cut issue, but once it starts to get so some complex that it all fragments, numbers become unrepresentatively small very quickly. And that's without considering how the question was worded, how the sample was collected, etc.

I don't know if we really need "lizardshit" as a broader term for the problems specific to opinion polls, or if we can just get away with calling it a variety of bullshit. On the other hand, polls are so common, perhaps it would serve as a reminder not to take them on face value. Lots of people are undeniably very, very stupid... but generally nowhere near as much as most polls seem to suggest. And that's probably worth remembering.