The (Greek) Tragedy Of The West

I try to avoid reading polemics because I don't see the point of them. I see the need to write them from time to time, but reading them is the cynical equivalent of mental masturbation. Sure, wallow in self-pity if that's your thing, but I'll stick to actual porn like a normal person, thanks.

Ahem.

Anyway, this is not to say I don't occasionally read things that sound more polemical, if only as a "just in case" situation. Today's IAI piece certainly has a "you should feel like utter shit" vibe about the title... I mean, "The End of the West means even more than you think" sounds to me like barely-coded essay-speak for "Things are awful and you're not feeling depressed enough, here, watch me murder some kittens so you feel properly sad."

Wait ! Don't go ! I promise it actually turns out to be nothing like that at all. It's actually about the similarities between the seemingly-chaotic Greek tragedies and the complex modern world in which we find ourselves, and how we can strive to keep our heads when those around us are losing theirs (and not necessarily just metaphorically). It's also good not just as a self-help, but an insight into what makes the Greek tragedies so appealing.

Still, it does begin with a bit of a rant about how things are bad :

Liberal humanism can be defined as a quasi-religious faith in progress, the power of reason, universal human rights, the moral primacy of Western-style democracy, free markets, and the unbounded potential of technology. It also contains a hubristic belief that it is the best single way in which human beings can organize themselves. It is a utopian ideology that looks forward to a time when all societies would be democratic, prosperous and part of a homogenous global system.

The gap between what liberal humanism promised and what it is now delivering is too vast for any illusion to sustain it. While technology has advanced the moral progress of our institutions has not followed. A backlash was inevitable.

Unfortunately, the author doesn't really expound on why things haven't panned out the way they were supposed to, and for my part I would put a pretty big label on "under-regulation". I don't think this has much to do with "liberal humanism" per se. I mean, monarchy is a terrible system, but if you happen to have a decent one, things are probably not too bad for the ordinary folk. Here it's the reverse : the ideas are good, but if you don't do things properly, if you let market forces and inequality spiral out of control, you won't get a good result.

Still, the main point seems sound. Rightly or wrongly, the perceived failings of the prevalent ideology are leading to reactions against it.

This backlash has been exploited by Trump and other right-wing politicians across the West who mask their self-serving politics by positioning themselves as bulwarks against the fragmenting of Western values. The utopias these new preachers evangelize about are now located in some unspecified time in the past, before the fragmentation, rather than the future. 

This is very interesting to me, seen as how medieval rulers were apt to present novelty as regression; elsewhere it has been noted that techbros sometimes try to disguise political regression with a shiny novel aesthetic. Here it seems we're really going for old-school conservatism : not merely avoiding further changes, but the notion that the past was better because it was geared around order and hierarchy, a direct reaction against liberal democratic values. The different ways people appeal for change are interesting in themselves, however the article continues :

Those still looking forward, ignore the fact that our uses of science and knowledge are as fleeting and crooked as we are ourselves, and pin hope of salvation on technology. They put their faith in the God-like abilities of AI or promise redemption in virtual worlds. Science can lift man’s eyes up to the stars, but humans will use what they know to satisfy their most pressing needs and desires — even if the result is ruin. 

While today’s techno-optimists may eventually develop the technology to get us to Mars, the technototalitarian colonies they found are unlikely to give us the belonging and meaning our society on our current planet isn’t providing.

Well that's grim ! But then, I've made my position on techno-optimism clear previously, so I'm not going to go through that again. As far as the Silicon Valley techbro mentality is concerned, I agree with the current article : these sorts of fools don't understand that technological advancement only brings benefits when coupled with social progress, and doesn't happen automatically. I wouldn't agree that this means technological advancement can or should be avoided, although I don't think the author is trying to imply that.

Despite their beliefs in gods and the afterlife, the ancient Greeks were focused on what one does in this world, in the here and now. When Odysseus visits Hades in the Odyssey he is approached by the masses of the dead. Speaking with Achilles he is told those that have crossed over the river Styx are “Numb dead people… the shades of poor exhausted mortals”. The afterlife is no substitute for the vividness of life itself.

And so, perhaps, are the purely-technological solutions of Silicon valley. The techbro solution is only a shade of what must be done to mitigate dystopian angst. An AI is not a true companion and never will be, and a virtual world is no solution to real problems. This doesn't mean they're innately bad, because they're not. But they are definitely used badly by the techbro types.

I would also add a brief note on dystopias more generally. It seems to me that just as utopian thinking commits a fallacy by imagining that an ideal system will lead to true, blissful perfection, so polemics against dystopias do the exact opposite. They paint this picture of a system in which everyone is miserable about everything the whole time, giving the impression that no other state of existence is then possible. All will suffer and all will be suffering. This is simply not how humans work : people will sometimes be happy almost regardless of circumstance, and equally, they can still be depressed in paradise. 

The thing that really gets me about polemics about the actual state of current affairs is the insistence that only one moral-emotional state is permissible : you should either be overjoyed at the ongoing technological progress, say, or utterly distraught and in tears about your constant oppression and the fact that everyone is trying to harvest your organs, or something. Other states are deemed invalid and your contrary opinion doesn't count; moral nuance is heresy. Only suffering or ecstasy are permitted. I hate this.

But I digress. Or do I ? Not quite. It's precisely this flawed, binary, good-versus-evil depiction which, the author suggests, the Greek tragedies are so good at avoiding :

A key set of treasures within this that perhaps best describes the world we now find ourselves in are the first written stories in Western culture – The Iliad and Odyssey – and the Greek tragedies. They describe a world in constant conflict defined by capriciousness, ambiguity, uncertainty and unknowability, where people are often swept along by forces, they interact with but don’t fully comprehend. They also then suggest how best we might approach such a world, pointing us to a form of both political and individual realism.

And that resonates with me very strongly. Yes, parts of the modern world are awful, and parts of the ancient world were orders of magnitude worse. But insisting that we hate everything continuously is just paralysingly nihilistic, not productive. We don't have to go down the full Stoic route to accept that we can still have a largely pleasant life despite circumstances.

One of the reasons the tragedies are still so relevant is that humans, despite advances in technology and material wealth are still driven by the same conflicting needs and illusions. There are many ways open to us to express our needs and organise ourselves, but we are limited by the same Hamartia or basic human vulnerabilities.

We live today in a supposed democratic meritocracy, yet our actions are subject to the invisible gods of market forces and algorithms. We can find ourselves pawns in the petty squabbles of autocratic leaders. But at the same time, within the confines of fate, we are free to make choices – this is how our destiny plays out. In perhaps the finest of the tragedies, Sophocles’ Oedipus Rex, Oedipus’ fate is not just decided by the choices he makes, but also the interaction of forces outside of his control interacting with his choices. 

Indeed, the developments of the modern world provide material improvement but often moral uncertainty. Whereas the natural forces of the ancient world made for permanent uncertainty, here it's the pace of our own discoveries which result in confusion : the effect is similar despite a very different underlying cause. 

And of course, we're none of us completely bereft of influence and control, but like even the heroes of Greek myth, we can sometimes feel very small and insignificant. The Greek heroes never really go on world-saving missions like modern superheroes do, that's left to the gods. But they do manage to make a difference in their own spheres. Surely that's an important lesson even for the most ardent would-be modern reformers : you may never fully realise your visions, but your contribution may be important even so.

The moral aspects, though, are where things get very interesting indeed :

These ancient stories show us that tragedy is not the triumph of evil over good, but the suffering caused by the triumph of one good over another. Tragedy is about morally defensible but incompatible goals... The chorus divides into two groups. There is no reconciliation.

Which is not to say that the modern world, and ancient tragedy, don't have elements of good and evil about them, things with which there can be no compromise. But an awful lot of conflict does arise in such a way. The similarities to the Dune saga, in which solution after solution is presented and found wanting, seem obvious. And to a lesser degree, perhaps, also that of certain fairy tales with no clear moral lesson (or indeed physical cause for why things happen*). Presumably, uncertainty is a necessary condition for dystopias, but maybe there's a glimmer of hope in that most good dystopian narratives nevertheless have a clear resolution, that form of "political and individual realism" mentioned earlier.

* Possibly, in those cases the chaos is the result of hidden variables : the original versions may have made a lot more sense in a context which is now lost to us. Both Greek mythology and later fairy tales are arguably ways to try and understand a world which simply couldn't be understood by the methods of rational inquiry available at the time.

What can we learn from this ? The author suggests :

The Greeks show us that killing and dying for concepts that defy reason, for often nonsensical ideas, is how many human beings have made sense of their lives. This is something those trying to negotiate peace today must not forget, but the unreasonable nature of our species is something we should all be aware of in our day-to-day relations. Tragedy teaches us that if we do not realise this and we try to disavow the past, then at best, we will be destined to repeat what has come before, and at worst, be destroyed by it. 

In essence, while you might be able to fight off irrationality to some degree, you can't eliminate it completely. You have to accept that people have irrational motivations and work within that framework, not treat them as machines who've simply been fed wrong data. 

The ancient stories place these dilemmas in rich historical and social contexts, where the past weighs down on the present, with everyone aware of the social networks in which they interact and the resulting duties and responsibilities this entails. This contrasts with the ahistorical approach taken by the hubristic planners of the invasions of Iraq and Afghanistan that tried to force an alien political construct on ancient cultures they made little effort to understand or the architects of the homogeneous globalized economy that uproot industries and move them round the world severing social and economic links in place for generations.

And I would add that this may also be a reason why less bloodthirsty culture wars can have such potent effects even on those who aren't directly affected by them. People are very deeply attached to their own beliefs and ways of doing things, especially when they don't ever question them. When someone tells you that something you've taken for granted is actually morally wrong when you didn't even think it was a moral issue at all*... it can feel like a direct attack. 

* Examples : which operating system to use, which food to eat, which bathroom to go to, how to dress, what TV channels to watch... and the left is as absolutely as guilty as the right at indulging in this sort of nonsense. 

Faced with the reality that there are moral and political dilemmas for which there are simply no solutions... recognises that there is a plurality of human values that determines many ways of living, and these values – and those that hold them – will inevitably clash. It is the search for a way of living together despite this, embracing the multiple forms of human life as a good thing in itself.

And for me the answer remains firmly in classical liberalism, a.k.a., you do you. When we interact, we both get a say in how things proceed. When we don't, then my actions are up to me and me alone, and for you as well. I know of no better proposed method of coexistence.

---

Provisionally, I remain unconvinced that "liberal humanism" has failed. An insufficiently liberal life, one in which everything we do was under someone else's direct control, would be a true dystopian hell, but I'm unconvinced that anything like that is on the horizon in the western hemisphere. I just don't see it. I often look around me, on a nice sunny day with the green grass and birds chirping and data reduction running happily in the background, see most people being generally pretty cheerful most of the time... and I wonder what the hell is wrong with the people on my social media feed. Important though it is to guard against backsliding, wallowing in self-misery that things don't meet our personal exacting standards, that everything is awful because there are some things we don't like, posting nothing much except what's wrong with the world rather than what's right, is a highly toxic lifestyle choice. 

All that said, perhaps it's all worth remembering the earlier lesson : that Blade Runner style grimdark dystopias are simply unrealistic. The important point is not to market everything as unbearably depressing, because it clearly isn't, nor is it that there's a slippery slope towards a truly hellish nightmare where there's no escape from torment, because that isn't going to happen either. I'm sorry, social media, but it just isn't, and this perverse sales pitch is massively ineffective for social reform because it's so blatantly at odds with reality. Rather the lesson here should be that just because there's much in the world that's joyous, that's no reason to excuse the very real failings, the times when things do get dark and depressing, the atrocities and outrages which do happen. We don't need any hyperbole about how we're all going to die horribly. Far more important is to tell us what we need to do to make improvements, not relentless doomongering.

My final point is that the idea of learning to embrace uncertainty, to "live more comfortably with doubt", is nothing new. Life has always been complicated and there have always been forces beyond our control we have to learn to accept; there are, after all, famous clichés about it I need not repeat. But recent decades do seem to have involved a genuine uptick in uncertainty and unpredictability, not just a perception of one. The author's point that this aspect is what makes the tragedies continuously appealing, and now even more relevant, is I think both novel and correct.

And yet the value of the Greek tragedies is, in a satisfyingly meta way, limited. They resonate precisely because they don't contain proposed solutions. The Greek heroes might succeed for a while, but never achieved anything lasting, just as our political efforts can also feel like hollow victories. So while they might teach us how some self-help lessons about how to face uncertainty, we cannot look to them for any deeper answers. Where we do face problems were a compromise is not possible, and this undeniably does happen, we must look elsewhere.

LLMs Ex Nihilo

WHAT'S IT ALL ABOUT ? SERIOUSLY ? WHEN YOU GET RIGHT DOWN TO IT ?

– Death, Soul Music (Terry Pratchett)

That's the scale of the problem we're facing when dealing with the issue of whether computers can think. Forget the robot uprising crap, we're talking full-on "meaning of life stuff" here. Allow me to explain.

As I've said previously, I've made peace with the idea that we'll probably never get a "truth engine" that can analyse data and form a truly objective, impartial conclusion. We might well get something that does better than we do at formulating conclusions though, which is not to be sniffed at. So instead I'm comfortable with describing AI (mostly but not entirely meaning LLMs) as "thinking engines" instead. That is, they process data and produce conclusions, and that's all I mean by "think". I'm using the word in a very broad, very liberal sense.

I tend to object rather strongly to the notion that LLMs currently go any way beyond this. I don't think they're conscious because I don't think they have any sort of subjective inner awareness. They're not really deterministic but they are still essentially algorithmic robots. They can be described as having a kind of understanding in a useful, productive sense, but not at all in the human-like way. They don't form mental images. They have no sense of the ineffable about them, no emotions, no desires of their own, they are nothing but data processors – and I think there's a very great deal more in a genuine mind than merely analysing raw input data.

The following article from Vox is remarkably thoughtful for a popular piece with a clickbaity headline. It's worth reading in full, but there are two essential points I want to highlight. The first is that minds are not necessarily all alike :

Radically different materials can execute the same basic operation. Biology may have produced the first flying entities. But the reason that birds can soar above the treetops isn’t that they’re made of organic tissue — it’s that their wings perform a set of aerodynamic tasks, such as generating lift and minimizing drag. As airplanes vividly demonstrate, if you put metal and fuel together in just the right way, you can replicate these functions and take to the skies.

To be sure, biological neural networks and artificial ones aren’t identical in design or behaviour. But neither is a cardinal [presumably meaning the bird, not the bishopy sort] and a Boeing 747. Nonetheless, the airplane replicates the avian functions that are necessary for flight. Likewise, computational functionalists wager that computers can instantiate all the neural operations that are relevant to consciousness. So, as long as they recreate a brain’s elaborate algorithms with sufficient precision, they actually can be conscious.

I will gloss over that "be" for the moment, because I take the point that minds might be qualitatively different from each other. Indeed, it would be surprising if they were not, animals having enormously different biology, senses, appendages, and environments with which they have to interact. So I'm prepared to concede that a conscious LLM may have little enough in common with the mind of a bumblebee or a dog or a professor of geology.

But how fundamental can this difference be before we stop calling it a mind at all ? For sure, it could be so different that we have very great difficulty in recognising an alien or robotic intelligence as being conscious. It seems to me, however, that if it doesn't have subjective, inaccessible, inner awareness, if it can't imagine things and those imaginings have no direct correspondence in the real world (just as we can imagine yellow without yellow being present), then we have no business calling it a mind, really. Thinking ? Understanding ? Yes, but only in the (still useful !) sense of data processing. Not at all in the philosophical sense. I see no reason whatever to presume that a network of probabilistic word generation algorithms, however elaborate, would count as truly having a mind any more than would an abacus. 

The article then explores (quite nicely) the notion of computational functionalism, wherein it's the process that's conscious, not the thing itself. I'm sympathetic to this view. Indeed, this could be interpreted as far more subtle than ordinary materialism or physicalism, in which some physical thing actually is conscious. You could even see it as a hybrid step, as close to dualism as a physicalist interpretation would allow*. And in that sense, I don't think it's crazy.

* That is, an electron is a real physical thing. But the mere movement of an electron isn't a thing. You can't say that movement is a physical object, because that's nonsense. So the process of electrons moving around, in a complex EM field, as being conscious is not really the same claim as saying the electrons themselves are actually little bits of minds.

The article's second major point is where I come unstuck :

We can name the physical laws that enable birds to get off the ground. And we have always had reason to believe that inanimate objects could emulate their movement; grains of sand have travelled through the air since time immemorial. By contrast, no one has ever seen a rock experience pain or pleasure, even momentarily (in part, because it’s impossible to directly observe the internal experience of any being or entity other than oneself).

There's the crux. You can't observe consciousness from the outside because it only exists internally, and whether this is because it's a process or an actual substance makes no difference at all. If you see a bunch of neurons firing, do you see yellow ? No, a bunch of neurons firing doesn't look yellow. If you see hormones diffusing throughout the brain, can you honestly say, "yep, I'm witnessing anger ?". No, you cannot. You have no idea what's going on in the conscious mind of the subject. 

The physical and the conscious clearly have some deep, intimate connection, but they are absolutely nothing like each other. There's a world of difference between saying that one gives rise to the other (I'm perfectly comfortable with that) and saying that the two things are literally the same.

A possible escape route is that maybe we can't say a person is angry or horny or introspective from a mere brain scan due to sheer complexity. Maybe we need to know everything in great detail, and we just can't capture the full neurochemical state in enough resolution to make more than a crude guess as to what a person is thinking. Perhaps if we could do so, we would literally be able to read minds.

I claim that this is not the case. I think the Hard Problem is actually the Impossible Problem. I claim that even if we had absolutely all physical information about the system, we would still never know what a person was thinking, not really.

Imagine, if you will, a stupendous spreadsheet listing the configuration of every atom, every subatomic particle in a human body. Could you use this to learn an awful lot about brain states, even a reasonable guess as to what the person was thinking about ? Yes, certainly. But no list of numbers ever actually is yellowness or being horny. The numbers [255, 255, 0] may describe perfectly to a computer how to produce the colour yellow, but it makes no sense to say that that list actually is yellow. They give rise to it, and that is all they do, all they can do.

"What ? That argument is stupid, I'm seeing [255, 0, 0] right now !" said no-one ever. You cannot experience a colour just from a list of numbers : they might give rise to your internal state, but that's not at all the same as saying they actually are the colour itself. That would be insane. Complexity is simply not the issue here at all.

Which brings me to the "meaning of life stuff" I promised earlier. What's been running around in my head is the opposite case : the idea that connections and processes are literally all there is. In that case, you could certainly say that an LLM had a similar consciousness to a human. You wouldn't have to resort to panpsychism, because you could legitimately say that only certainly processes were complex, and not everything in existence undergoes such processes.

But the result would be, I think, extreme nihilism of such a degree as to be actually offensive. I've postulated that meaning arises through knowledge of the connections between different things, that we can be said to understand a thing based on how much we know about how it relates to other things and our ability to predict its behaviour in novel situations. The more moral sense of the word "meaning" may encode something similar : meaningful experiences are those which affect many of our internal links, our own local knowledge of how things relate to each other.

Now suppose that those connections are all there are. That is, knowledge, understanding, and ultimately consciousness itself, arise from connections between items of information and nothing else. That's it. We've hit rock bottom. A mind is a set of relations between physical things and there's nothing else to it. Mystery solved, we can all go home for tea and biscuits.

Or, quite honestly, we may as well all do whatever the fuck we want, or just not bother. I find this idea unspeakably bleak, because if that's all there is, then we're just... electrons and magnetic fields. Atoms bashing about. We have no more claim to moral value than a rock. Without something deeper than the visible connections (some continuing series of connections, possibly going all the way down), we could not rightly say that murder or torture was wrong, and that I simply will not accept.

"But,", you may say, "maybe the moral meaning is something that emerges from the system if it can't be found within it reductively". Ah yes, you might say that. But that concedes the point that there's more to the system than what's visible, doesn't it ?

Fortunately for my own mental health, I think this idea has absolutely no merit in it. The idea that a mind literally is the observable processes, or the physical constituent parts of the nervous system, is to me absolutely and incomprehensibly WRONG. 

Now to be sure many people accept the materialistic world view without falling into moral degeneracy; some at least have probably even fully embraced the "it's all just atoms" perspective without becoming absolute cunts (though I would be a wee bit skeptical as to just how many have really thought this through). And that's fine, I guess, as long as it helps them get through the day. But I cannot escape the feeling that for me personally, this doesn't work. If other people want to live in such a reality, then good for them, I suppose. But for me ? I think I'd rather hit myself in the face with a brick, thanks. After all :

If droids could think, there'd be none of us here.

– Obi-Wan Kenobi, Star Wars

The Wisdom of Russell Howard's Grandmother

Today's post resumes my unusual habit of amateur epistemology.

I've explored the definition of understanding many times, concluding that it's the knowledge of how things connect and interrelate. The more we understand something, the better we can predict how it behaves in novel situations. Fair enough as far as it goes, but I've always found the main issue with this is what happens when we reach some seemingly irreducible fact that we can't understand. We can always memorise a mathematical operator, however complex it might be, but that's not at all the same as being able to apply it in anger.

Here I can offer two explanations for why we reach such limits. The first I can suggest immediately, while the second will take a bit longer and be developed over the rest of the post.

The first explanation is simply hardware. It may be that we just can't hold more than a certain number of connections to some mental objects, just as we can't process things at arbitrarily high speeds. Perhaps the brain allocates only a certain volume to each topic, and when we reach its limit, we simply can't add anything more into that particular bucket. It might be that either we just cannot absorb anything more on a broad subject*, or that an apparently singular individual item just requires too many connections to too many others for us to properly understand it – in essence, a straw that breaks a camel's back*.

* Like when Homer Simpson forgets how to drive because he takes a wine-making course.
** Perhaps somewhat literally. I had several lecture courses which imparted negative knowledge, meaning I had less understanding of the subject than when I went it.

This, I think, gets us a long way, but still doesn't fully address the more fundamental limits of understanding. For that, I'm going to pursue the more philosophical issue of wisdom. Even Plato never really came up with a convincing definition of this, so you can't accuse me of lacking ambition.

It's probably helpful here to recap the last time I examined such issues. In that post I ventured four main ideas :

• Analytical thinking asks, "what if this is true ?", exploring the full consequences of a proposition.
• Critical thinking asks, "is this actually true ?", being a concern for accuracy rather than with exploring any consequences.

Intimately connected with these two main points were two other slightly more amorphous concepts :

• Curiousity is the yearning for more knowledge. It can take different forms, such as the desire to learn about more and more topics (e.g. consuming endless Buzzfeed lists) versus the desire to verify existing claims, but the essence of it is the same.
• Multi-level thinking is the ability to consider a position on different scales, e.g. whether each line of code is syntactically correct versus whether the underlying method is doing what it's supposed to be doing. Grammar Nazis versus fact-checkers, I guess.

All of these are closely related, and separating them like this is somewhat artificial... but, as we shall see, useful.

Which all leads to my proposed definition of wisdom : knowing the best thing to do. 

Hmm. That seems a bit trivial to bother with.

A slightly less compressed form might help : knowing how to carry out the best solution to a problem. But this is probably still too compressed to seem of any use, so let's deconstruct this more fully. It's been carefully phrased to include two key aspects. First, the wise thinker must be able to assess a proposed solution and realise if there's a better approach. But secondly, the alternative they suggest must be something they actually know how to enact. After all, there's no wisdom in realising that everyone would be happier (say) if you gave them all more money unless you have a workable scheme to raise the necessary funding.

This definition works, I think, for both moral and purely logical problems. To give a recent example of the latter : I gave ChatGPT a coding question, saying I'd found a particular method to solve my problem which should work, but it explained that there was a much better approach so it went off and implemented that instead (in this case it worked perfectly – and this was a problem I'd previously spent some weeks trying to figure out from first principles*). 

* For the interested reader : I wanted to use binary_fill_holes to fill in meshes in Blender with an integer grid of vertices. ChatGPT realised that there was a much better, though badly-documented, Blender-internal solution that was the ideal way to meet my objective. With hindsight, my own solution was actually pretty darn close, but the specific implementation was full of holes... pun intended, sorry not sorry.

Morally, the obvious case is Jurassic Park. Sure, you could bring back dinosaurs, but that famously doesn't mean that you should. Or as Russell Howard says, sure, legally you can wake up your gran while dressed as Hitler, but that doesn't mean it's a good idea.

In some sense this could be described as supercritical thinking. It's concerned not just with all scales of the problem itself, but goes beyond that to consider its full consequences in context, to address whether the proposed solution would be a good approach or even whether the problem is one that needs solving at all. It's a union of critical, analytical, and multi-level thinking all combined and expanded. Rather than knowing what sort of thinking to apply, as I tentatively suggested previously, wisdom might be better described as turning rational thought up to 11.

And this takes us back to the limits of understanding. Wisdom here might be in recognising that these limits simply cannot be broken, that trying to probe any deeper won't result in anything useful – that we should stop when we have a definition that's actionable and have shown that further investigation won't bring any more improvements*. Likewise, I could go further with this post to better define what I mean by "best solution". But this would open up an enormous can of worms that probably wouldn't help and would make everything much longer. The wiser course of action seems to be stop here as far the definition goes. What remains is only a few clarifications and some practical consequences.

* In this particular case, we end up trying to define words by using other words, and hit the limit of what pure language can convey. I have some further musings on this which I may or may not get around to writing up eventually.

Wisdom, like intelligence, is sometimes used as a synonym for raw knowledge. If instead we say it means knowing what's the best thing to do, then clearly this requires knowledge, just as it needs both critical and analytical thinking skills. But it's not the same as any of these. We can immediately see that the correlation could be imperfect, that someone might have a huge breadth and depth of knowledge but be unable to see the relevant similarities across different fields.

This strongly suggests that wisdom is a thing that can be taught... at least, to the same extent that knowledge can be taught. The behaviour of LLMs, as per the earlier example, might offer some clues here. For these, I strongly suspect that knowledge and wisdom are strongly coupled, that all you need to make them wiser is a larger training data set and a bigger context window – they'll consider more information from more fields of expertise at more and more scales automatically*. That said, you can't really "teach" an LLM anything except by fully retraining it, which in effect gives you a whole new model. All you can really do is instruct them.

* Though of course, the quality of the output still depends on the quality of the input training data as well as the prompt.

This is not much like humans, who can certainly be taught and indeed are (mostly) capable of learning from their mistakes. Indeed, wisdom requires knowing what to avoid just as much as when to proceed. Whereas in an LLM wisdom may emerge naturally as a function of size of training data, this is (I think) likely only a trivial result of that training data containing more and more wise behaviour. It's much harder to gauge whether this happens for humans through absorbing sheer volume of knowledge, though I suspect if we're only ever told, "these things are true, learn these parrot-fashion" instead of, "here's how to evaluate knowledge", the result tends to be someone who's neither wise nor critical. Our own training certainly does matter.

Of course, LLMs don't really have beliefs and opinions in the same way that humans do. An LLM is a mass of statistical information and probabilistic weights, with no real fixed ideas at all – certainly not between conversations in the same way that humans have some ideas that they hold as almost permanently fixed. But likewise, updating our own world view in response to new information is seldom easy, just as including it in an LLM isn't as simple as telling it something in a single conversation. The analogies are interesting just as much as for their differences as their similarities.

In any case, the ability to update one's world view is not the definition of wisdom, but it does follow directly from it. The wise thinker knows when a single fact is of limited consequence and when it may necessitate a paradigm shift. They are able to judge when the new information is itself likely wrong and when it's their own existing ideas which are at fault. They consider also the metadata of who said it and why – they do not evaluate it purely on its own merits*. So the ancient Greeks were right to value to self-knowledge, as understanding one's own biases is essential in understanding how we respond to new data, but this isn't wisdom in itself, just as the ability to learn from experience is part of it, but not itself the definition of wisdom.

* The idea that ignoring the source of information is somehow actually the correct, rational approach is a curiously persistent and incredibly widespread error. 

Finally, it's obvious how the Jurassic Park scientists were intelligent but unwise. Russell Howard, by contrast, is much less intelligent but much wiser : having  a really dumb idea but realising that this would be a Bad Thing To Do. What of his grandmother ? Well, if she wakes up convinced that Hitler has returned, she's not very wise at all, but if she realises that this is so incompatible with her well-established knowledge that Hitler is long dead, then probably she's a lot wiser than her grandson. So c'mon Russel, put it to the test. For SCIENCE !

The Logician's Swindle

What makes a puzzle annoying ? When is solving a problem rewarding, and when is finding out the answer just frustrating ? If we could answer this, we might get a long way towards making the world a happier place. Getting people to actually enjoy solving problems, rather than being pissed off at their opponents for discovering a flaw in their arguments, would surely benefit political discourse enormously.

I don't propose to try and answer all of this today. Instead, what I can do is address one particular aspect of the problem. I say that at least one major cause of puzzles being annoying rather than enjoyable is when you've been outright cheated, and that this happens far more often than it should.

Specifically, consider Newcomb's Paradox as described on Veritasium. The video begins :

You walk into a room, and there's a supercomputer and two boxes on the table. One box is open, and it's got $1,000 in it. There's no trick. You know it's $1,000. The other box is a mystery box, you can't see inside.

Now, the supercomputer says you can either take both boxes, that is the mystery box and the $1,000, or you can just take the mystery box.

So, what's in that mystery box?

Well, the supercomputer tells you that before you walked into the room, it made a prediction about your choice. If the supercomputer predicted you would just take the mystery box and you'd leave the $1,000 on the table, well, then it put $1 million into the mystery box. But if the supercomputer predicted that you would take both boxes, then it put nothing in the mystery box.

The supercomputer made its prediction before you knew about the problem and it has already set up the boxes. It's not trying to trick you, it's not trying to deprive you of any money. Its only goal is to make the correct prediction.

So, what do you do? Do you take both boxes or do you just take the mystery box?

Don't worry about how the supercomputer is making its prediction. Instead of a computer, you could think of it as a super intelligent alien, a cunning demon, or even a team of the world's best psychologists. It really doesn't matter who or what is making the prediction. All you need to know is that they are extremely accurate and that they made that prediction before you walked into the room.

I highlight certain parts because they feel crucial. To me, this is saying very explicitly, "don't think about this aspect of the problem, it's not important at all". Were this not so, I would otherwise object to how such a thing could be possible, and the details would certainly matter : was the machine running over a diverse sample of people, or was there something particular about them that helped its accuracy ? But no, this apparently isn't important, so whatever misgivings I have about free will and suchlike, I willingly surrender for the purpose of the test. I put them aside, still fully expecting to be fooled (I suck at logical puzzles) but in some other way.

Having made that assumption, the answer is obvious. If the machine is essentially always accurate, I take one box. It knows, magically, that this box will contain a million dollars, and I walk out happy and rich and in search for a bank offering a good exchange rate to a proper currency. 

But later in the video we get :

Here's how I think about the problem in a way that actually makes sense. You know that the supercomputer has already set up the boxes, so whatever I decide to do now, it doesn't change whether there's zero or $1 million in that mystery box, and that gives us four possible options that I've written down here.

If there is $0 in a mystery box, then I could one-box and get $0 or I could two-box and get $1,000, but there could also be $1 million in a mystery box. And in that case, I would get $1 million if I one-box or I would get $1,001,000 if I two-box. So, I'm always better off by picking both boxes.

Rubbish. Complete twaddle. You just told us that the machine is accurate and we shouldn't factor this in to our calculations, but in this way of thinking you cannot possibly ignore how the machine works. This is not even self-consistent ! By saying that the machine is essentially perfectly accurate, you've eliminated the very possibility of $1,001,000. That can only happen if the machine actually is inaccurate in some cases, which to my mind you've all but told us directly to discount.

This, then is a swindle, and one common to various logical puzzles. "Don't think about this aspect of the problem", they say, only later to say, "Hah ! You should have thought about this aspect of the problem after all, you fool !". Right, so you expect me to think you're a liar ? How is that a fair test ?

The rest of the video is a perfectly decent discussion of free will etc. (Veritasium is one of my favourite YouTube channels), but the poor description from the outset makes the whole thing a mess. Having been told that accuracy was not an issue, I expect something else I've overlooked to come into play. Naturally I overlooked determinism and all that because you told me to overlook it. The pettiness of it all annoys me quite intensely.

Don't worry, I'm not going down the free will avenue with this post. Rather, I just want to briefly outline that this kind of swindle is common to logic problems, and is itself one particular expression of a more general reason they're so often very irritating.

The closest similarity is surely the Monty Hall problem (the one with the prize goats). That one always confused the heck out of me because people never properly explained that I should have been paying crucial attention to the host's knowledge, not how many goats there are or how many doors. But any logic puzzle can suffer if you're not properly informed about what the key aspect of the problem is, or worse, if you're actively told to ignore it.

Not that framing doesn't sometimes reveal something very interesting. Wason's selection is fascinating in showing how the same people can have much more difficulty solving the same task if it's described slightly differently – especially so when the alternative form is nothing they wouldn't also be familiar with. But there, the whole point is to study psychology. No deception is employed, no swindle pulls the solution out from beneath the solver's feet. The facts are laid bare and it presents a straightforward yet surprising challenge to many people who take it. No, framing is only annoying when it's done to deliberately thwart the participant. 

There's also a common tendency for the puzzle-setter to declare the rational solution from authority, saying "this is obviously the correct solution because the alternative doesn't make sense to me". A classic example concerns people refusing small amounts of compensation when they would normally expect a much bigger payout. Time and time again we hear people declaring that accepting the small offer would be rational since they come out with a net cash gain. But to any sensible person there are a multitude of reasons why this would be an extremely foolish thing to do : accepting the initial offer may deny them any chance at the larger amount; they may simply feel insulted and disrespected, and responding to such behaviour is essentially letting the bully get away with it. It is only rational in an incredibly narrow and naive economic sense, and more broadly simply isn't rational at all*.

* Veritasium does this with a unique peculiarity, openly acknowledging that the "irrational" decision of choosing one box is the more profitable. I find this is going deep into "what's wrong with you ?" territory.

Again, this is a sort of swindle, denying the opposing argument by forbidding debate rather than engaging with it on an equal footing. You thought things were going to be fair and above-board only to find out that they were anything but, that the answer had already been decided without you.

Another similarity is the pettiness. Veritasium didn't have to pull the rug out from under the viewer's feet any more than anyone has to accept that getting a smaller payout is somehow rational. 

Very occasionally, I've run public surveys to help me with my own research. I've tried to ensure the wording was extremely careful, including omitting details when this would bias the result. For example I once ran a public poll on how many groups of points – galaxies – people could see in a plot, deliberately not telling them what they were looking at. Some people objected that there wasn't enough information (e.g. what sort of scales they should be considering), and I sympathise that they might find this annoying. But for me this was the whole point, to gauge what people's natural reactions were : I wanted to know if they would instinctively identify the same groupings that appeared natural and obvious to me (most of them did, as it turned out). I needed to know if my additional knowledge was biasing me, or if the groupings I identified would be readily visible without this extra information. 

The point here is that there's absolutely no reason for misdirection. It's perfectly possible to account for this in a way that will give you a meaningful result to the question you're asking. Sometimes, this can only become apparent after the fact, but in those cases the participant should feel relieved, not annoyed. Annoyance only happens when the misdirection was unnecessary. 

A second personal example : group meetings back in my PhD days. These served the valuable purpose of getting the students used to dealing with tough questions. But they also turned the experience into a weekly grilling that made the whole thing quite intensely annoying... instead of having an enjoyable, low-stakes discussion about science, we had to deal with supervisors being deliberately over-critical. That we all knew full well what was going on didn't help in the least. It would have been fine if such sessions had been clearly demarcated and set aside as such, with regular meetings more about science for its own sake. Trying to pretend this was how scientific discussion should happen, though, was just unfair.

Again, there was no reason for the misdirection. This too was a sort of swindle. Oh, you think you're here to discuss your work ? You thought I was being harsh because I wanted to be ? Hah hah, fooled you ! The idea that maybe they could have just not done that was never raised.

On an grander scale, problems with the alternatives to dark matter. This too feels like something of a swindle : proponents often raise objections to dark matter which are based entirely on the properties of the ordinary matter we can see. They make highly dubious inferences about the necessary connection to the dark matter they're trying to demonstrate doesn't exist, saying that the lack of a naively-expected correlation proves it can't exist. Some of these problems can become obvious, but sometimes it's worth spelling this out at the high level because it's all too easy to lose sight of the forest for the trees. Once you start questioning the underlying assumption and realising that maybe the connection isn't so direct after all, often the whole thing falls apart.

And in other arenas too we find possible swindles. As I've covered before, thought experiments become extremely annoying when changing a small detail would profoundly alter the result but the instigator refuses to consider any variation : no, you must focus on this aspect of the problem because I said so, even if my scenario is actually bunk. Just like insisting someone should accept a miniscule payout, it's disrespectful not to think the other person's opinion might have some value. 

Likewise with analogies. An indirect analogy can be extremely powerful when the relevant aspect is sufficiently similar to its comparison subject, becoming thought-provoking in both its similarities and in its minor, extraneous differences. When an analogy is intended to be direct, though, the seemingly-extraneous details can become crucial, so expecting people to shut up and ignore them is not realistic. It's extremely difficult to focus on the "relevant" bit (usually declared by authority) when there are obvious deficiencies in the whole thing. Conversely, it does no good to pretend similarities don't exist when they do, or to overlook them on grounds which are actually minor details or only quantitative differences.

All this sets out some conditions for when puzzles becoming annoying, and gives us a rough working definition : The Logician's Swindle is the use of unnecessary misdirection from a position of unjustified authority.

This is similar to but not quite the same as the Magician's Choice. In the latter, we know we're being denied crucial information, misdirected, and otherwise deceived. We go in with eyes open knowing we'll almost certainly be tricked and often paying for the privilege of suspension of disbelief. We know we won't be able to solve the problem and we enjoy our failed attempts to work out what's going on.

The Logician's Swindle is altogether nastier. Here, we're supposed to have all the information we need to reach the "correct" conclusion, but we find only afterwards that actually we don't – with the swindler often denying this for the sake of making us look foolish. And the conclusion itself may be open to dispute but the proponent argues from a completely artificial authority that it isn't. Worst of all is that "mistakes" can (though do not always actually) carry real-world consequences. In short, it's a scam : a discussion that should be in good faith which actually isn't.

And that's why I hate logic puzzles.

I'd Like To Teach Machines To Think

Today, a couple of contrarian pieces claiming that maybe LLMs do think and reason after all.

That is, not in a namby-pamby, "it's just something similar enough to thinking that we might as well call it that" sort of way. This is perfectly reasonable. I stand by that myself. To get hung up on saying "they're not really thinking" every time someone casually uses this instead of "processing data" is frankly just annoying, not productive. Likewise for intelligence : if they're taking input data and producing coherent output, well I call that a form of intelligence at the very least.

No such linguistic sleights of hand are to be found here though. No no, these pieces are much closer to the dreaded C-word... consciousness.

The first article makes much the weaker claims of the two. This one touches on the self-awareness issue, but its main point is simply that they're doing something more than pure word prediction.

Modern LLMs (Claude, GPT-4, and others) have an interesting feature, the humble thinking/reasoning tokens. Before generating a response, the model can generate intermediate tokens that the user never sees (optional). These tokens aren't part of the answer. They exist between the prompt and the response, modifying the context that the final answer is generated from and associated via the attention mechanism. A final better output is then generated.

Every token between the prompt and the response is, in information-theory terms, an opportunity for drift. The prompt signal should attenuate with distance. Adding hundreds of intermediate tokens into the context should make the answer worse, not better. But reasoning tokens do the opposite. They add additional machine generated context and the answer improves. The signal gets stronger through a process that logically should weaken it.

This process improves performance. That thought should give you pause, just like how a thinking model pauses to consider hard problems before answering. That fact should stop you cold.

As someone who's not in any way, to any degree, expert in LLMs, this is highly unintuitive. After all, when I start thinking deeply, I don't expect that the more I think, the more off-track I'll go and the worse my output will become. I expect that the more things I take into consideration, the more deeply I delve into the problem, the more accurate and insightful will be my final output. I guess I presume LLMs are doing something broadly analogous in that they are (in effect) considering ever-more voluminous amounts of text in producing their final response after a chain-of-thought. The more they consider the relations between different words, the better should be their final output.

I asked ChatGPT about this, and its (considerably shortened) response was as follows :

I think the article is smuggling in a dodgy intuition there. The right baseline is not “extra tokens should obviously make things worse”. In a transformer, later tokens are not forced to overwrite earlier ones in a simple serial way. Attention lets the model keep referring back to whatever parts of the prompt or scratch work are useful, so intermediate text can genuinely help by turning one hard prediction into several easier ones. That is exactly why scratchpad methods and chain-of-thought prompting often improve performance on multi-step tasks.

So I would rephrase the issue like this : intermediate steps should not be expected to degrade performance in general. They should be expected to help when they function as a good scratchpad, and to hurt when they function as self-generated noise. The article seems to assume that the mere presence of extra tokens ought to be harmful. That is too simple. The real trade-off is not between “direct answer” and “more text”, but between “useful decomposition” and “error propagation”. One small clue that this trade-off is real is that CoT can also make models more confident when they are wrong, which is exactly what you would expect if self-generated reasoning sometimes stabilises mistakes instead of correcting them.

This seems not crazy. It then seems too much of a stretch in the article, to me, to claim that because the model is reasoning "in the context of a probability distribution", it's still doing something directly (and I emphasise the emphasis here most emphatically) analogous to some aspects of human reasoning. I think we have the capacity for a much deeper, truer understanding than any LLM has or ever will have.

If you wish to reduce this to "just" token prediction, then your "just" has to carry the weight of a system that monitors itself, evaluates its own sufficiency for a posed problem, decides when to intervene, generates targeted modifications to its own operating context, and produces objectively improved outcomes. That "just" isn't explaining anything anymore. It's refusing to engage with what the system is observably doing by utilizing a thought terminating cliche in place of observation.

None of this requires an LLM to have consciousness. However, it does require an artificial neural network to be engaging in processes that clearly resemble how meta-cognitive awareness works in the human mind. At what point does "this person is engaged in silly anthropomorphism" turn into "this other person is using anthropocentrism to dismiss what is happening in front of them"?

This doesn't feel warranted to me. For sure, humans probably use linguistic heuristics in place of "actual" reasoning more than we like to, err, think. LLMs manipulating text, in ways not matter how arbitrarily complex, does not indicate any evidence of actual, true reasoning and thought. It's just an incredibly clever way to predict words, and I see no evidence of anything deeper going on at all. No sentience, no awareness, no emotions, no preferences, no inner light. The LLM literally does not exist when it isn't prompted. It has no consciousness, no subconsciousness, no self of any kind.

The irony if course delectable... I prefer the LLM's claim that it isn't thinking to the human's assertion that it is !

Before I go to the next article, I also have to mention a recent discussion with a very interesting claim indeed :

The thing is, Yann LeCun was actually right. Purely text-based LLMs never learned that if you push a table, you also move the objects on the table. What happened instead is that LLMs became “multi-modal” and made to accept images, audio, and video as input as well as text. So “AI” did learn that if you push a table, you also move the objects on the table, but Yann LeCun was right that they did not learn it purely from text.

Despite this coming from a genuinely proper expert, I struggle to believe it. It simply doesn't match my experience with LLMs at all... well, maybe a little bit with GPT-3.5, but even with that hilarious dumbass, only a little bit. Causal connections between objects don't seem particularly difficult to establish via pure text : if you move a table, you move all objects on that table... and LLMs are surely very good at knowing what word represents a literal object. This really doesn't feel like something that should present any difficulty. 

I wish 3.5 were still with us... I'd love to test it.

GPT-5.4 says that this claim isn't correct, but that there is an interesting point behind it. That is, multi-modal models do help LLMs learn things that humans never bother writing down, but that "text-only models clearly do learn a fair amount of everyday physical regularity from language". This I would definitely believe. Without some very specific documentation though, the claim they can't learn something as basic as objects being moved when their supporting table moves is something I'd be very reluctant to concede. The current "car wash" problem is an interesting reminder that LLMs aren't human, but not, I think, proof categorical that they're totally lacking in any sort of reasoning capacity whatseover.

On to the second, much more full-throated article.

The fundamental case against the “I” in AI is that intelligence is organic, derived from sensory interaction with a physical environment. Agüera y Arcas turns the tables with the premise that computation is the substrate for intelligence in all life forms. The claim builds on an apparently crude proposition: prediction is the fundamental principle behind intelligence and “may be the whole story”.

I react quite instinctively against this, essentially with, "fascinating, tell me more about your stupid idea !". That is, there are some things I think are gloriously weird. I love their sheer audacity, may or may not hold them respectable, but don't believe them for a microsecond. I do not mean "stupid" here in an especially pejorative sense, but if you can't already understand it, then I probably can't explain it.

A central tenet of What is Intelligence? is that every form of life is an aggregation of cooperative parts. Links proliferate through patterns that enable increasingly complex functions. When Agüera y Arcas says the brain is computational, it’s not a metaphor: it is not that brains are like computers, they are computers.

He is erasing a familiar conceptual boundary here: intelligence does not prompt function, it is function. Intelligence, he argues, is a property of systems rather than beings, and function is its primary indicator. A rock does not function, but a kidney does. This is demonstrated simply by cutting them in half. The rock becomes two rocks, but the kidney is no longer a kidney.

So does a kidney have intelligence? Or an amoeba? Or a leaf? These questions are opened up, along with the question of whether Large Language Models have intelligence, which may a better way to frame it than asking whether they are intelligent.

In another discussion, I could not make myself understood when I tried to say I think that awareness is something one has, not what one is. This is extraordinarily hard to explain if you don't already "get it", but cannot for the life of me understand the claim that experiences are literally the same as physical brain states. To me this is an absolute non sequitur, with the evidence being possibly the clearest that could ever be presented for anything ever. I won't try and do it again – this blog is chock-full of that kind of stuff as it is – but maybe it's still useful to frame how I think about LLMs.

I do not have any issue with humans eventually constructing some sort of true AI. I think it's perfectly possible we could construct a chip or something which would give rise to (or otherwise access) consciousness. I do not think an LLM will ever do that, because it's literally just rearranging text. I see no more reason an LLM could be conscious than the text of a newspaper if it were cut to pieces and thrown into a whirlwind. This is why that when I say it makes good sense to describe LLMs as intelligent and reasoning, I do so with a quite monumental proviso that this in no way whatsofuckingever implies they do so much like humans – at least, not with regard to the full scope of how humans think.

 Maybe some bullet points would help ? Well, let's try. I claim :

• LLMs can be said to think and reason in that they process input data to produce sensible outputs. In certain domains, they can already do so with an accuracy that rivals and exceeds humans.
• LLMs function as more than pure word predictors; they can generalise and abstract to a useful degree, though highly imperfect and not on a par with humans except for the stupid ones (of which unfortunately we must contend with many). They do some things with appreciable, meaningful similarity to humans.
• LLMs are nevertheless just mechanical. They don't have an inner life, are incapable of feeling anything, have no desires, no sensations, and no awareness of even the smallest degree. You can't program a true AI, you have to build one. BUT...
• ...who on Earth would want a mechanical mind that would potentially suffer or try and eat us or whatnot ? Pointless. Far better to stick with an LLM-like route and go for pure tool development; if you want to bring new souls into the world, there's exactly one way to do that, and it ain't about building robots in your garden shed.

There, does that clear things up ? I doubt it. This has been the most decoherent of Decoherency posts, so if you think I'm being self-contradictory, you're wrong and I don't care. But I feel slightly better, anyway.

Listening To The Voices In Other People's Heads

Here's a very nice long read from the Guardian about trying to understand what's really going on inside our heads.

Recently I concluded that any attempt to understand what consciousness actually is is likely hopeless. Trying to understand what we mean by experience when literally all we have access to is experience is inevitably circular. But the effort itself isn't fruitless. We can all of us have different preferences for what we think is going on – non-physical, spiritual, purely materialistic – and that discussion is often productive, if only in understanding how people reach radically different conclusions from the same data. More promisingly, it gives us a better handle on how we go about defining things, how we grapple with the imperfections of mapping language onto reality.

But there were also two more directly productive outcomes described in the Aeon essay I looked at last time. One was that we could understand in some detail the neural correlates of consciousness, the processes occurring within the brain that are associated with what we think and experience. The second, somewhat subtler issue, was that we can still take a reductive approach to different aspects of consciousness : we can describe it in terms of different levels and content, and in so doing get back to something we can discuss in familiar scientific terms. Just as we don't have to worry about what a quark is really made of to understand how it behaves, so too we can tackle the subject of minds.

The Guardian piece is complementary to the Aeon article in that it leans more heavily in this direction. As it begins :

A neuroscientific perspective on consciousness might tell us something about its neural correlates, but it is unlikely to tell us much, if anything, about the nature of thoughts or the textures of inner experience; it’s the wrong tool for that job. So what might we learn about consciousness if we gave more weight to the view from inside the experience – the phenomenological viewpoint ?

For example, it describes William James' comments on something I've found extremely strange for many years :

“Suppose we try to recall a forgotten name,” he writes. “The state of our consciousness is peculiar. There is a gap therein; but no mere gap. It is a gap that is intensely active.” A sort of ghost of the absent name haunts the empty space in our consciousness, he suggests, making us “tingle with the sense of our closeness, and then letting us sink back without the longed-​for term”.

He goes on: let someone propose a candidate for the missing name, he suggests, and even though we have no consciousness of what the name is, we are somehow conscious of what it is not, and so summarily reject it. How strange! Our consciousness of one absence is completely different from our consciousness of another. But, he asks, “how can the two consciousnesses be different when the terms which might make them different are not there ?” 

The feeling of an absence in our minds is nothing like the absence of a feeling; to the contrary, this is an absence that is highly specific and intensely felt. Thoughts glimpsed from some height of awareness but somehow not yet formed, much less put into words or images – this is the subtle terrain James invites us to explore with him.

I very much agree with this. It's always seemed weird to me how we can think in complete sentences. We – or rather I, and I'll get back to why the distinction matters later on – don't really sense the words falling into place, they just come out like that. Clearly they must have been assembled at some point, but somehow this happens without us knowing about it ! 

Perhaps even weirder is that sensation when grappling with a complex problem : at the point of reaching a possible solution comes a very distinct sensation of raw, unstructured thought, some quasi-awareness that the answer has been reached but without being able to articulate it. That moment is crucial. If interrupted in this momentary phase, the proto-thought may be lost entirely. But if it's seen through to completion, the thought crystallises into language : something we can easily memorise, recall, and communicate with others.

Much of the article is then concerned with whether we can access this much lower level of thinking in some way. If we want to understand consciousness, can be go beyond language ? Maybe that's too ambitious. To start with, can we at least access thoughts at the stage that they become coherent ? 

Step forth, Russell T. Hurlburt :

For half a century now, Hurlburt has been scrupulously collecting reports of people’s inner experiences at random moments – and just as scrupulously resisting the urge to draw premature conclusions. A die-​hard empiricist, he is as devoted to data as he is allergic to theories... I’ve been going around with a beeper wired to an earpiece that sends a sudden sharp note into my left ear at random times of the day. This is my cue to recall and jot down whatever was going on in my head immediately before I registered the beep. The idea is to capture a snapshot of the contents of consciousness at a specific moment in time by dipping a ladle into the onrushing stream.

What he is after in his research is the “pristine inner experience”, by which he means a sample of human thought “unspoiled by the act of observation or reflection”. Like James, Hurlburt acknowledges that the act of recalling and describing an experience is bound to alter it, but he believes that his method can get us closer to the uncontaminated ideal than any other.

In some ways this should be extremely easy. Again, we definitely do have coherent, structured, linguistic thoughts, and writing those down is straightforward enough. Indeed, as this fascinating BBC article describes, it's even possible to read these directly from the brain. This doesn't require participants to mentally try and speak : to a degree, thoughts can now be extracted at a lower level than this.

But then again, even the most fully-developed thoughts can be extremely slippery. As I've noted, once you put pen to paper you engage in a sort of self-conversation, thoughts and beliefs becoming highly flexible once you start reflecting back on them from an external input. Take a thought from in here and put it out there and it inevitably changes, even if only just a little. 

Still, this can largely be avoided : once a sentence is formed, it can be written down. A more difficult aspect of the problem is trying to disentangle that bit of coherency from everything else we're thinking about :

I took out the little pad provided by Hurlburt and jotted down this thought: “Deciding whether or not to buy a roll.” I know, not terribly exciting, but it seems very few of my mental contents are. I was thinking ahead to lunch and wordlessly deliberating whether to buy a fresh roll for a sandwich or do the responsible thing and use up the heel of bread I had at home. I was also conscious of the pattern of the skirt – an unflatteringly large plaid – worn by the woman standing in line ahead of me. 

Was that observation part of the moment in question, or did it come immediately before or after? I couldn’t say for sure. (How long does a moment in consciousness last ?) And what about the pervasive smells of freshly baked goods and cheese ? These both preceded and followed the moment under examination, but were they present to my awareness at the beep ?

Throw in just a few complications and suddenly the problem becomes much more difficult, maybe even impossible. Words ? We can attend to them. The whole facet of experience, including our different senses, how they affect us, when they occur, when we assemble a sentence ? That's much more fuzzy. Trying to describe anything before the point of coherency may even be a non sequitur. Maybe we can pin down a bit more about the process by determining what we're currently experiencing (e.g. which senses are given priority over the others, where and when we give out conscious attention to language rather than sensory experience, what counts as thought, etc.) but there will be some limits as to how far we can get with this.

Some really fascinating things have come out of the research though :

The first finding, to which I can personally attest, is just how little most of us know about the characteristics of our own inner experiences. “That’s probably the most important finding that I’ve got,” Hurlburt said.

Important, yes, but I think the other findings are a lot more interesting : 

Inner speech, which many of us – including many philosophers and neuroscientists – believe is the common currency of consciousness, may actually not be all that common. Hurlburt estimates that only a minority of us are “inner speakers”. So why do we think we talk to ourselves all the time? Perhaps because we have little choice but to resort to language when asked to express what we are thinking. As a result, we’re “likely to assume that’s the medium for inner thought”.

But that doesn’t make it true for everyone. Fewer than a quarter of the samples that Hurlburt has gathered report experiences of inner speech. A slightly lower percentage report either inner seeing, feeling, or sensory awareness. Still another fifth of his samples report experiences of “unsymbolised” thought – complete thoughts made up of neither words nor images. Hurlburt has suggested that we fail to recognise the diversity of thinking styles because we lump them all together under that single word – thinking – and assume we mean the same thing by it, though in actuality we don’t.

Aphantasia and the lack of inner speech is something I've covered many times before, but this is something beyond that. It's something I know I must have but am absolutely incapable of imagining. A truly pure thought consisting of... what, exactly ? Not language. Not any of the senses. Just pure electrochemistry, I guess. That's absolutely wild. 

EDIT : To a degree, I can imagine this. My inner monologue is pretty incessant, but it's not constant. There are times when it shuts up and all I have is sensory experience and emotions. What I absolutely cannot do is properly articulate what's going on. Is there still some processing going on using language at the lower levels, ready to be raised to my awareness for perusal when required ? Or is it fundamentally different at the earlier stages and only converted to language later on ? More below.

I wonder if the brain scans described in the BBC would be capable of interpreting these in the same way as for the (to me normal) case of thinking with an inner voice or eye. Perhaps it's like blindsight. That is, maybe our brains are all doing basically the same low-level stuff, but sometimes not everything is raised to whatever part it is that brings it to conscious awareness. Or maybe, even more interestingly, we don't all work in quite the same way. Regardless, scans of people who aren't thinking with inner speech, imagery, or any kind of structured thoughts would surely make for a fascinating comparison. Would the scan reveal the same thing as in those with well-defined internal monologues or would it show something else altogether ?

Another researcher suggests a different and more holistic approach :

The field’s focus on conscious perception has led it to overlook the 30-50% of mental experience that is fed to us by our minds rather than our senses, Kalina Christoff Hadjiilieva contends. “Consciousness is just one function of the mind,” Hadjiilieva told me during one of a half-​dozen interviews, this session over a cup of tea in my garden. “To focus on conscious thoughts is like focusing on the leaves of a tree and trying to understand them in isolation,” she said. “The tree is the mind, and there’s a lot more to the mind than consciousness.”

The degree to which the mind wanders appears to be surprisingly important :

Hadjiilieva conducted an experiment with long-​term meditators (mindfulness practitioners). These are people who have been trained to still their minds but also to notice the precise moment when that stillness is broken by an errant thought, which Hadjiilieva found happens every 10 to 20 seconds or so even in these trained minds. (“The big lesson of meditation,” she said, “is that the mind cannot be controlled.”)

This makes intuitive sense to me, and again maybe reveal something about the structure of thought processes. The way I like to work even, when in a state of relative focus, is often to flit back and forth between a couple of different things at once. I like to check my emails and glance at the news quite frequently, only going into really deep focus every once in a while*. WIth some tasks this works very well : it's like I have my brain keep working on the other thing in the background while giving my consciousness time to rest by attending to something easier. What's crucial for me, though, is that these must be activities of my own choosing. Being disturbed by an external influence is a big no-no, If someone interrupts me then the process is instantly broken.

* Though my work habits vary considerably. For code I nearly always concentrate on the code and absolutely nothing else, with a similar situation for most difficult problems. It's for the routine, less cognitively demanding tasks that I prefer to have multiple tabs open, as it were.

This all ties is quite nicely to the earlier discussion :

Hadjiilieva and her colleagues noted a jump in activity within the hippocampus, a key component of the default mode network that is involved in not only memory but also learning and spatial navigation. To their surprise, the leap in hippocampal activity preceded the arrival of the thought in the meditator’s consciousness by nearly four seconds – an epoch in brain time, and far longer than it takes for a sensory impression to cross the threshold of our awareness.

You might wonder if this further shifts my uncertainty about the apparent non-physical nature consciousness. In this case, it doesn't. I've already covered similar experiments regarding free will, and here it seems to me that no neural correlate could be anything remotely like subjective experience : how do some electrons whizzing about resemble the smell of a daffodil or the feeling of anger ? They simply don't, and to assume otherwise is to completely miss the point of the Hard Problem. But to build on from the previous post, this is still very interesting stuff : 

“Something is going on prior to awareness,” Christoff Hadjiilieva said, but she’s not sure exactly what it is or why it takes so long. This finding indicates that a spontaneous thought must undergo some sort of complicated unconscious processing before finding (or forcing) its way into the stream of consciousness. For Hadjiilieva, the mystery she’s uncovered points to what she regards as the “really hard problem of consciousness” – how the contents of the unconscious form into thoughts that sometimes find their way into our awareness, and sometimes don’t.

Well, that's definitely a hard problem, and maybe it would even be better to call it the hard problem. The Hard Problem of the philosophical sense may well turn out to be the Impossible Problem : we literally can't understand our own subjective experience, since by definition this is all we have access to. In that case some relabelling makes good sense. It seems very reasonable that the time delay points towards the brain doing some unconscious information processing before raising it to our awareness, and understanding how and why this happens seems extremely difficult but far from outright impossible.

The article concludes, in typical Guardian fashion, with a warning of the dangers of capitalism in preventing our minds from going about their productive, unguided wanderings, as well as the difficulties of persuading people to treat research into the subconscious as serious science. Perhaps the author should have read that BBC story. If you can access this with a machine, the danger may not be that nobody takes it seriously, but the exact opposite.

The Truth About Utility

What makes a useful definition ? Originally I had a much more philosophically pretentious post semi-drafted for this, and I may still do that one separately. But various recent discussions have taken me down a very different path, one which might be more, err... useful. So let's start with this one and see if I ever get back to the nature-of-reality version in a future post. 

A good definition, I think, must surely be something which is widely applicable but also specific : things which happen frequently but not always. It should be readily distinguished from similar counter-examples. Crucially, it cannot be something which either never happens or always happens. It shouldn't be something which forbids itself entirely or makes it inevitable or ubiquitous. It should describe a specific thing that actually sometimes happens, or is at least conceptually valid and distinct from other, similar terms. 

What I see people doing is trying to make things true or false by setting their definition up in such a way that it cannot ever fail, and that to me seems like a mistake. This doesn't mean we can't have productive discussions, but it does, I think, impose some extremely unhelpful limitations. 

Let's do this one by example. First I'll look at cases where people define things such that they can't ever happen, and then the reverse, related case of defining them such that they're inevitable. Both in my view are counterproductive mistakes. They are terminology problems but they prevent us from getting at what we really mean, which is usually much more interesting.

1) Defining things out of existence

If we define something with such precision, such high standards, or such that it involves a logical contradiction that it can't ever be true, then I submit that this isn't a useful definition at all. Furthermore, it's likely not what we really mean when we use the term in everyday discourse.

Malevolence : Plato and other ancient philosophers help that nobody would knowingly do evil. I forget who it was who described it explicitly (possibly several people), but the basic idea was that if you knew something was wrong, you couldn't possibly do it. You might still carry out an immoral action, but you'd be misjudging, thinking that the gratification you would get would outweigh any negative consequences there might be for anyone else. Alternatively, you might do so only because you hadn't realised the existence, extent, or nature of those negative consequences.

I think this is a deeply mistaken view of humanity. As per the link, people certainly carry out heinous acts in full knowledge of their full consequences, sometimes this being the very reason for their behaviour rather than a side-effect. Or they may know but simply not care. But they aren't, I think  carrying out a mental calculus of where the balance lies. Even if they were, this would still make the word – or notion of wilful harm – meaningless. The point for most discussion is that people cause each other harm sometimes because they want this to happen, not out of ignorance. Anything more beyond this rapidly leads into such convoluted nuances that the definition collapses into uselessness. 

Or to put it another way : "Sure, he committed the murder, but it wasn't out of malice : that's impossible, so he must've done it because he mistakenly thought his pleasure at the victim's suffering would outweigh their actual suffering". 

To me this makes the word unproductively useless, trying to define the thing out of existence. Surely that points towards this meaning not being what we truly meant : the important thing is that people inflict harm on others for its own sake, and inferring anything further is best avoided altogether.

Altruism : In the opposite case, my partner likes to say that nobody is really altruistic. Everyone acts, she says, because they believe there will be some benefit to themselves, even if that reward is purely emotional. In the extreme case, someone might be give their own lives to save others, not because they thought the value of the lives they saved outweighs their own, but because of the fleeting emotional reward they themselves will get from knowing the others will live.

This too I think is surely putting more on the word than it can bear. The point of altruism, I'd say, is that we sometimes value others more than ourselves and act to bring a net benefit to them even at the expense of our own status. Start demanding that we get no emotional reward at all and again the term has been defined out of meaningful existence. This makes it utterly useless, and surely, therefore, this can't be what most of us mean most of the time when we use it. I'll qualify this a bit more later, but that general-case point is the one I want to focus on.

Knowledge and understanding : I've covered the nature of LLM-outputs several times, most recently here but also e.g. here and (tangentially) here. More on those more directly in a minute, but a closely related claim is whether they can be said to truly understand anything. I think they can, in the carefully qualified sense that a) they have access to some form of information; b) they form connections between different pieces of information; c) they act in a logical, coherent way to predict how things behave in novel situations. Not perfectly, it's true, but more interesting by far is that they do it at all. 

Now for sure, this is not the same sort of understanding that humans have. But its qualitative similarities, in my view, outweigh and are far more interesting than the quantitative differences between silicon and neural understanding. I think it's just not at all useful to say that "meaning only comes from humans ascribing this to the output". This is so inevitably necessary that it adds nothing useful to the discussion : well who else was going to be reading the output then ? And if you define "understanding" to be only a human thing, then it's tautologous that no non-human will ever have it. That's cheating.

2) Defining things into existence

We can now see how the reverse is also true : if we define a thing as being completely unavoidable, we won't get anywhere.

Hallucinations : See the links in the previous entry as this follows directly from the previous definition. I did initially agree that it was sensible to describe all LLM output as a hallucination, but I changed my mind some time ago. Given that they are now able to process complex (and multi-modal) information in a way that closely aligns with human expectations, and can in fact exceed our own predictive capacities at least some of the time, I now no longer think describing their output at purely hallucinatory makes much sense. 

It's more useful, I think, to say they're hallucinating (in their own peculiar way) when their output has no connection whatever to the input data or prompt. This is much more analogous to human hallucinations in which we see things which aren't there. I would still agree, provisionally, that LLMs treat all information has having a much more similar level of validity than humans do, and undeniably they have some qualitative as well as quantitative differences from human thought. But they are very clearly not purely fabricating stuff all the time : more often than not, they're processing their inputs quite sensibly.

Importantly, the claim that all LLM output is a hallucination is consistent with the notion that they don't understand anything. I'm not claiming incoherency here : I'm claiming that these definitions should be discarded because they aren't useful, not because there's any inherent problem as such. The alternative definitions I've suggested are, I think, better only because they are more flexible and specific, allowing us to describe things in more detail, not because they eliminate any inconsistency.

God : Don't say I'm not ambitious ! The old argument that god is necessarily perfect and perfection necessarily exists... well, surely this is the ultimate case of trying to assert truth by definition. God is a perfect what, exactly ? A perfect square ? A perfect teapot ? Well, if a perfect teapot exists, where is it ? Could it be Russel's Teapot, somewhere beyond Earth's orbit ? Surely not, because if it was perfect, it would be in my hands whenever I need it. But it isn't, and therefore the perfect teapot clearly does not exist. 

And if even the perfect teapot doesn't exist, I see no reason to say that the abstract concept of perfection itself – a Spinozan notion of God – also has to exist in any sense beyond a mental construct. Clearly, I can imagine what I think a perfect teapot should be like, but that has no further existence outside my head. There's no reason to think that perfection itself is any different.

So here too, "perfect" in the everyday sense does not mean the same thing as St. Anselm would have it mean. Nobody uses perfect to mean "something which must exist" : indeed, we often use it to describe things which can't exist precisely because they're perfect ! "Platonic ideal" might be one of Plato's better ideas here, if only in the concept : we can conceive of better examples of chairs and circles and virtues even if we can't bring them into being. That's generally how we use the word, to describe something specific in aspect, not the singularity-like God of the Upanishads. 

As far as the existence of God goes, and very much with my agnostic hat one, I think definitions here are of no help whatever. We can conceive of perfect examples of things we fundamentally do understand, like circles. But perfection itself ? That would require understanding all facets of existence, which as imperfect beings we simply can't do and never will. A general understanding of perfection is beyond our limitations : we can no more say that "god's perfection means he exists" than we can say what a perfect dinosaur would be like. The concept may simply be incomprehensible or it may not even make sense at all.

---

That's my idea of a good definition then. It should be specific, flexible, distinct from alternatives, and describe things which occur at a finite rate (even if only conceptually). If a definition forbids itself from ever existing, or would always be true, then it has no use cases and should be discarded. Those kinds of definitions usually twist readily-comprehensible everyday meanings into something convoluted, unproductive, and useless.

I'll stress "usually" a little bit though as I don't say that the extremes don't matter at all. For example, what do we mean when we say we're know something or that we're certain of it ? Usually, that our own belief is well-formed and our confidence is beyond reasonable or routine doubt. We don't usually mean that we have found Truth Itself, that we can state our claim with literally zero chance of it being wrong, and that all unbelievers are evil and/or stupid. 

Like the case of purest selflessness, this kind of concept definitely does have value, but more in the philosophy classroom than the real world. The extreme cases let us frame our own actual beliefs and compare them to those of others, rather than providing useful, workable definitions in themselves. For example, we can all agree on what true certainty would actually mean, but to use the word more practically, we have to scale things back. That's where the discussions start to get interesting, trying to figure out the limits of our own underlying reasoning as well as that of the others in the debate. 

To a very large extent, I think the question of how we use a definition is very much the same as what we think it means at the most basic level. But then, others may have a different understanding. I don't always agree with the alternatives, but trying to figure them out is usually the fun bit.