A quantitative method of knowing who to trust

An excellent, fascinating insight into quantifying self-knowledge and metaknowledge as a means of weighting opinions. Via Benjamin Ljung.

One reason that crowds mess up is the hegemony of common knowledge. Even when people make independent judgments, they might be working off the same information. When you average everyone’s judgments, information that is known to all gets counted repeatedly, once for each person, which gives it more significance than it deserves and drowns out diverse sources of knowledge.

The solution, Prelec suggests, is to weight answers not by confidence but by metaknowledge: knowledge about knowledge. Metaknowledge means you are aware of what you know or don’t know, and of where your level of knowledge stands in relation to other people’s. That’s a useful measure of your value to the crowd, because knowledge and metaknowledge usually go together. ‘Expertise implies not only knowledge of a subject matter but knowledge of how knowledge of that subject matter is produced,’ says Aaron Bentley, a graduate student at the City University of New York Graduate Center who studies social cognition.

Whereas you might have no independent way to verify people’s knowledge, you can confirm their metaknowledge. When you take a survey, ask people for two numbers: their own best guess of the answer (the ‘response’) and also their assessment of how many people they think will agree with them (the ‘prediction’). The response represents their knowledge, the prediction their metaknowledge. After you have collected everyone’s responses, you can compare their metaknowledge predictions to the group’s averaged knowledge. That provides a concrete measure: people who provided the most accurate predictions – who displayed the most self-awareness and most accurate perception of others – are the ones to trust.

Sociologists have long relied on a version of this approach, asking people not just what they know but what they think other people know. In so doing, the researchers can gauge the prevalence of beliefs and activities that people won’t admit, even to themselves. It’s suspicious whenever people say an activity is common but claim they’d never – never! – do it themselves... Truly innocent people are inclined to think the best of others.

A strong consensus is the closest proxy to truth that we have. The lone wolf who knows better than the misguided masses is much rarer than Hollywood movies would lead you to believe. To take advantage of these principles yourself, you don’t have to do a formal survey; just pay attention to the metaknowledge exhibited by people around you. Experts are more likely to recognise that other people will disagree with them, and they should be able to represent other points of view even if they don’t agree. Novices betray themselves by being unable to fathom any position other than their own. Likewise, you can monitor your own metaknowledge by noting when you find that more people hold a certain belief than you expected. That doesn’t automatically make you wrong, but does suggest you should take another look at your beliefs.
https://aeon.co/essays/a-mathematical-bs-detector-can-boost-the-wisdom-of-crowds