Sam Altman Isn't A Nice Chap

Today, an excellent piece of investigative journalism from The New Yorker. 

Anyone remember those brief few days in 2023 when Sam Altman was removed from the board of OpenAI ? Of course you do. But now, more than two years later, we finally get an inside look into what the hell that was all about. Months of investigations have revealed in extreme detail just what was going on behind closed doors. If you're at all interested in these kinds of companies, then this is well worth your time. 

In the most compact form possible, this amounts to : "Sam Altman isn't a paedophile, but he's definitely a lying scumbag". As far as a character assessment goes, this is devastating. With regards to OpenAI the company, the picture is considerably more nuanced, especially in terms of the actual researchers involved (Altman himself is only a businessman). To their great credit, the author here interviews everyone involved, including Altman on multiple occasions. This is by no means a hit piece, but what emerges is abundantly clear. Virtually all those involved say that Altman is an inveterate liar, and while Altman may rightly dispute the interpretation of some recollected conversations, the idea that everyone else but him is misremembering beggars belief. 

In PDF form this runs to 39 pages. No doubt other media outlets will produce a more distilled version, but in the meantime, what follows below is my own 2200 word-summary in pure quotation form. To keep the narrative coherent, some of these are rearranged from their original sequence but no paraphrasing is used. Highlights are my own.

But first, there are two comments I should add for my own perspective. First, the main concerns of the other employees and board members revolved around the issue of AI safety, especially whether it would be aligned with human values and suchlike. I personally believe that these concerns are – provisionally – not sensible. I've said many times that while we can legitimately call LLMs intelligent in a carefully-defined way, I don't believe they show the merest glimmer of consciousness. Safety in terms of "killer robots will be the death of us all" is, I believe, complete hogwash; LLMs have absolutely no will of their own, and if putting them in certain situations is a bad idea, that is entirely the result of human error : we can simply choose not to do that, and nothing stops us from pulling the plug. I said almost from the very start that the whole "this is too dangerous" thing was a genius bit of marketing spin, not a genuine concern on Altman's part.

The provision I make here is that safety and alignment may still matter in the lesser regard of AI not behaving as expected. This certainly does pose problems and it clearly is important to have it behave in a basically-predictable way, as a useful tool rather than as an independent entity. But given a lack of any consciousness, I don't worry at all about a rogue AI deliberately trying to murder everyone of its own volition, as many of the top researchers are apparently genuinely concerned with. 

What matters here in terms of Altman's character is, then, not who's right about safety, but what he said he'd do about it. He lied about this over and over again, and whereas absolutely everyone tells lies from time to time, the degree to which Altman does this, routinely, is inexcusable. He's also a bullshit artist. It's fine to change your mind, but you should acknowledge that you've done this, as well as explaining how and why. Altman simply doesn't bother.

This leads me to my second point : people are being awfully naïve about this. The subreddits exploding with people saying they were ditching OpenAI for Anthropic never made any sense; the idea that the US government bunch of deranged fascist fuckwits wouldn't be using LLMs was beyond belief, never mind that Anthropic were already deeply in bed with the US military (the article also goes into Anthropic's misbehaviour as well, though to a lesser extent). Trying to lionise Anthropic for pushing back on a couple of points is, in my view, a classic example of naivety curving back on itself in an ouroboros of cynicism. And again, to pretend that a CEO will never lie is an absurdity, but this should not be the take-home message of this piece. Rather it should be the more realistic and far more important point that Altman lies so much you can never trust him about anything. That's what should concern everyone, not the non-story that "CEO is a scumbag". That's par for the course. 

That's more than enough from me. On, then, to the summary.

---

In a tense call after Altman’s firing, the board pressed him to acknowledge a pattern of deception. “This is just so fucked up,” he said repeatedly, according to people on the call. “I can’t change my personality.” Altman says that he doesn’t recall the exchange. “It’s possible I meant something like ‘I do try to be a unifying force,’ ” he told us, adding that this trait had enabled him to lead an immensely successful company. He attributed the criticism to a tendency, especially early in his career, “to be too much of a conflict avoider.” But a board member offered a different interpretation of his statement: “What it meant was ‘I have this trait where I lie to people, and I’m not going to stop.’ ” Were the colleagues who fired Altman motivated by alarmism and personal animus, or were they right that he couldn’t be trusted?

Most of Altman’s employees at Loopt liked him, but some said that they were struck by his tendency to exaggerate, even about trivial things. One recalled Altman bragging widely that he was a champion Ping-Pong player—“like, Missouri high-school Ping-Pong champ”—and then proving to be one of the worst players in the office. (Altman says that he was probably joking.) As Mark Jacobstein, an older Loopt employee who was asked by investors to act as Altman’s “babysitter,” later told Keach Hagey, for “The Optimist,” a biography of Altman, “There’s a blurring between ‘I think I can maybe accomplish this thing’ and ‘I have already accomplished this thing’ that in its most toxic form leads to Theranos,” Elizabeth Holmes’s fraudulent startup.

If everything went right, the OpenAI founders believed, artificial intelligence could usher in a post-scarcity utopia, automating grunt work, curing cancer, and liberating people to enjoy lives of leisure and abundance. But if the technology went rogue, or fell into the wrong hands, the devastation could be total. China could use it to build a novel bioweapon or a fleet of advanced drones; an A.I. model could outmaneuver its overseers, replicating itself on secret servers so that it couldn’t be turned off; in extreme cases, it might seize control of the energy grid, the stock market, or the nuclear arsenal. 

Not everyone believed this, to say the least, but Altman repeatedly affirmed that he did. He wrote on his blog in 2015 that superhuman machine intelligence “does not have to be the inherently evil sci-fi version to kill us all. A more probable scenario is that it simply doesn’t care about us much either way, but in an effort to accomplish some other goal . . . wipes us out.” OpenAI’s founders vowed not to privilege speed over safety, and the organization’s articles of incorporation made benefitting humanity a legally binding duty. If A.I. was going to be the most powerful technology in history, it followed that any individual with sole control over it stood to become uniquely powerful—a scenario that the founders referred to as an “AGI dictatorship.”

By September, 2017, though, Musk had grown impatient. During discussions about whether to reconstitute OpenAI as a for-profit company, he demanded majority control. Altman’s replies varied depending on the context. His main consistent demand seems to have been that if OpenAI were reorganized under the control of a C.E.O. that job should go to him. Sutskever seemed uncomfortable with this idea. He sent Musk and Altman a long, plaintive e-mail on behalf of himself and Brockman, with the subject line “Honest Thoughts.” He wrote, “The goal of OpenAI is to make the future good and to avoid an AGI dictatorship.” He continued, addressing Musk, “So it is a bad idea to create a structure where you could become a dictator.” He relayed similar concerns to Altman: “We don’t understand why the CEO title is so important to you. Your stated reasons have changed, and it’s hard to really understand what’s driving it.”

By 2018, Amodei had started questioning the founders’ motives more openly. “Everything was a rotating set of schemes to raise money,” he later wrote in his notes. In early 2018, Amodei has said, he started drafting a charter for the company and, in weeks of conversations with Altman and Brockman, advocated for its most radical clause: if a “value-aligned, safety-conscious project” came close to building an A.G.I. before OpenAI did, the company would “stop competing with and start assisting this project.” According to the “merge and assist” clause, as it was called, if, say, Google’s researchers figured out how to build a safe A.G.I. first, then OpenAI could wind itself down and donate its resources to Google. By any normal corporate logic, this was an insane thing to promise. But OpenAI was not supposed to be a normal company.

That premise was tested in the spring of 2019, when OpenAI was negotiating a billion-dollar investment from Microsoft. Although Amodei, who was leading the company’s safety team, had helped to pitch the deal to Bill Gates, many people on the team were anxious about it, fearing that Microsoft would insert provisions that overrode OpenAI’s ethical commitments. Amodei presented Altman with a ranked list of safety demands, placing the preservation of the merge-and-assist clause at the very top. Altman agreed to that demand, but in June, as the deal was closing, Amodei discovered that a provision granting Microsoft the power to block OpenAI from any mergers had been added. “Eighty per cent of the charter was just betrayed,” Amodei recalled. He confronted Altman, who denied that the provision existed. Amodei read it aloud, pointing to the text, and ultimately forced another colleague to confirm its existence to Altman directly. (Altman doesn’t remember this.) 

In the course of several meetings in the spring of 2023, Altman seemed to waver. He stopped talking about endowing a prize. Instead, he advocated for establishing an in-house “superalignment team.” An official announcement, referring to the company’s reserves of computing power, pledged that the team would get “20% of the compute we’ve secured to date”—a resource potentially worth more than a billion dollars. The effort was necessary, according to the announcement, because, if alignment remained unsolved, A.G.I. might “lead to the disempowerment of humanity or even human extinction.” 

The twenty-per-cent commitment evaporated, however. Four people who worked on or closely with the team said that the actual resources were between one and two per cent of the company’s compute. Furthermore, a researcher on the team said, “most of the superalignment compute was actually on the oldest cluster with the worst chips.” The researchers believed that superior hardware was being reserved for profit-generating activities. (OpenAI disputes this.) Leike complained to Murati, then the company’s chief technology officer, but she told him to stop pressing the point—the commitment had never been realistic... the superalignment team was dissolved the following year, without completing its mission.

By then, internal messages show, executives and board members had come to believe that Altman’s omissions and deceptions might have ramifications for the safety of OpenAI’s products. In a meeting in December, 2022, Altman assured board members that a variety of features in a forthcoming model, GPT-4, had been approved by a safety panel. Toner, the board member and A.I.-policy expert, requested documentation. She learned that the most controversial features—one that allowed users to “fine-tune” the model for specific tasks, and another that deployed it as a personal assistant—had not been approved. 

Last June, on his personal blog, Altman wrote, referring to artificial superintelligence, “We are past the event horizon; the takeoff has started.” This was, according to the charter, arguably the moment when OpenAI might stop competing with other companies and start working with them. But in that post, called “The Gentle Singularity,” he adopted a new tone, replacing existential terror with ebullient optimism. “We’ll all get better stuff,” he wrote. “We will build ever-more-wonderful things for each other.” He acknowledged that the alignment problem remained unsolved, but he redefined it—rather than being a deadly threat, it was an inconvenience, like the algorithms that tempt us to waste time scrolling on Instagram.

Some people defended Altman’s business acumen and dismissed his rivals, especially Sutskever and Amodei, as failed aspirants to his throne. Others portrayed them as gullible, absent-minded scientists, or as hysterical “doomers,” gripped by the delusion that the software they were building would somehow come alive and kill them. Yoon, the former board member, argued that Altman was “not this Machiavellian villain” but merely, to the point of “fecklessness,” able to convince himself of the shifting realities of his sales pitches. “He’s too caught up in his own self-belief,” she said. “So he does things that, if you live in the real world, make no sense. But he doesn’t live in the real world.”

Yet most of the people we spoke to shared the judgment of Sutskever and Amodei: Altman has a relentless will to power that, even among industrialists who put their names on spaceships, sets him apart. “He’s unconstrained by truth,” the board member told us. “He has two traits that are almost never seen in the same person. The first is a strong desire to please people, to be liked in any given interaction. The second is almost a sociopathic lack of concern for the consequences that may come from deceiving someone.”

Six people close to the inquiry alleged that it seemed designed to limit transparency. Some of them said that the investigators initially did not contact important figures at the company. Others were uncomfortable sharing concerns about Altman because they felt there was not a sufficient effort to insure anonymity. “Everything pointed to the fact that they wanted to find the outcome, which is to acquit him,” the employee said. 

Given OpenAI’s 501(c)(3) status and the high-profile nature of the firing, many executives there expected to see extensive findings. In March, 2024, however, OpenAI announced that it would clear Altman but released no report. The company provided, on its website, some eight hundred words acknowledging a “breakdown in trust.” People involved in the investigation said that no report was released because none was written. Instead, the findings were limited to oral briefings.

Many former and current OpenAI employees told us that they were shocked by the lack of disclosure. Altman said he believed that all the board members who joined in the aftermath of his reinstatement received the oral briefings. “That’s an absolute, outright lie,” a person with direct knowledge of the situation said. Some board members told us that ongoing questions about the integrity of the report could prompt, as one put it, “a need for another investigation.”

In a meeting with U.S. intelligence officials in the summer of 2017, he [Altman] claimed that China had launched an “A.G.I. Manhattan Project,” and that OpenAI needed billions of dollars of government funding to keep pace. When pressed for evidence, Altman said, “I’ve heard things.” It was the first of several meetings in which he made the claim. After one of them, he told an intelligence official that he would follow up with evidence. He never did.

“My vibes don’t match a lot of the traditional A.I.-safety stuff,” Altman said. He insisted that he continued to prioritize these matters, but when pressed for specifics he was vague: “We still will run safety projects, or at least safety-adjacent projects.” When we asked to interview researchers at the company who were working on existential safety—the kinds of issues that could mean, as Altman once put it, “lights-out for all of us”—an OpenAI representative seemed confused. “What do you mean by ‘existential safety’?” he replied. “That’s not, like, a thing.”

Altman is not a technical savant—according to many in his orbit, he lacks extensive expertise in coding or machine learning. Multiple engineers recalled him misusing or confusing basic technical terms. He built OpenAI, in large part, by harnessing other people’s money and technical talent. This doesn’t make him unique. It makes him a businessman. More remarkable is his ability to convince skittish engineers, investors, and a tech-skeptical public that their priorities, even when mutually exclusive, are also his priorities. When such people have tried to hinder his next move, he has often found the words to neutralize them, at least temporarily; usually, by the time they lose patience with him, he’s got what he needs. “He sets up structures that, on paper, constrain him in the future,” Wainwright, the former OpenAI researcher, said. “But then, when the future comes and it comes time to be constrained, he does away with whatever the structure was.”

Even people close to Altman find it difficult to know where his “hope for humanity” ends and his ambition begins. His greatest strength has always been his ability to convince disparate groups that what he wants and what they need are one and the same. He made use of a unique historical juncture, when the public was wary of tech-industry hype and most of the researchers capable of building A.G.I. were terrified of bringing it into existence. Altman responded with a move that no other pitchman had perfected: he used apocalyptic rhetoric to explain how A.G.I. could destroy us all—and why, therefore, he should be the one to build it. Maybe this was a premeditated masterstroke. Maybe he was fumbling for an advantage. Either way, it worked.

The Wisdom of Russell Howard's Grandmother

Today's post resumes my unusual habit of amateur epistemology.

I've explored the definition of understanding many times, concluding that it's the knowledge of how things connect and interrelate. The more we understand something, the better we can predict how it behaves in novel situations. Fair enough as far as it goes, but I've always found the main issue with this is what happens when we reach some seemingly irreducible fact that we can't understand. We can always memorise a mathematical operator, however complex it might be, but that's not at all the same as being able to apply it in anger.

Here I can offer two explanations for why we reach such limits. The first I can suggest immediately, while the second will take a bit longer and be developed over the rest of the post.

The first explanation is simply hardware. It may be that we just can't hold more than a certain number of connections to some mental objects, just as we can't process things at arbitrarily high speeds. Perhaps the brain allocates only a certain volume to each topic, and when we reach its limit, we simply can't add anything more into that particular bucket. It might be that either we just cannot absorb anything more on a broad subject*, or that an apparently singular individual item just requires too many connections to too many others for us to properly understand it – in essence, a straw that breaks a camel's back*.

* Like when Homer Simpson forgets how to drive because he takes a wine-making course.
** Perhaps somewhat literally. I had several lecture courses which imparted negative knowledge, meaning I had less understanding of the subject than when I went it.

This, I think, gets us a long way, but still doesn't fully address the more fundamental limits of understanding. For that, I'm going to pursue the more philosophical issue of wisdom. Even Plato never really came up with a convincing definition of this, so you can't accuse me of lacking ambition.

It's probably helpful here to recap the last time I examined such issues. In that post I ventured four main ideas :

• Analytical thinking asks, "what if this is true ?", exploring the full consequences of a proposition.
• Critical thinking asks, "is this actually true ?", being a concern for accuracy rather than with exploring any consequences.

Intimately connected with these two main points were two other slightly more amorphous concepts :

• Curiousity is the yearning for more knowledge. It can take different forms, such as the desire to learn about more and more topics (e.g. consuming endless Buzzfeed lists) versus the desire to verify existing claims, but the essence of it is the same.
• Multi-level thinking is the ability to consider a position on different scales, e.g. whether each line of code is syntactically correct versus whether the underlying method is doing what it's supposed to be doing. Grammar Nazis versus fact-checkers, I guess.

All of these are closely related, and separating them like this is somewhat artificial... but, as we shall see, useful.

Which all leads to my proposed definition of wisdom : knowing the best thing to do. 

Hmm. That seems a bit trivial to bother with.

A slightly less compressed form might help : knowing how to carry out the best solution to a problem. But this is probably still too compressed to seem of any use, so let's deconstruct this more fully. It's been carefully phrased to include two key aspects. First, the wise thinker must be able to assess a proposed solution and realise if there's a better approach. But secondly, the alternative they suggest must be something they actually know how to enact. After all, there's no wisdom in realising that everyone would be happier (say) if you gave them all more money unless you have a workable scheme to raise the necessary funding.

This definition works, I think, for both moral and purely logical problems. To give a recent example of the latter : I gave ChatGPT a coding question, saying I'd found a particular method to solve my problem which should work, but it explained that there was a much better approach so it went off and implemented that instead (in this case it worked perfectly – and this was a problem I'd previously spent some weeks trying to figure out from first principles*). 

* For the interested reader : I wanted to use binary_fill_holes to fill in meshes in Blender with an integer grid of vertices. ChatGPT realised that there was a much better, though badly-documented, Blender-internal solution that was the ideal way to meet my objective. With hindsight, my own solution was actually pretty darn close, but the specific implementation was full of holes... pun intended, sorry not sorry.

Morally, the obvious case is Jurassic Park. Sure, you could bring back dinosaurs, but that famously doesn't mean that you should. Or as Russell Howard says, sure, legally you can wake up your gran while dressed as Hitler, but that doesn't mean it's a good idea.

In some sense this could be described as supercritical thinking. It's concerned not just with all scales of the problem itself, but goes beyond that to consider its full consequences in context, to address whether the proposed solution would be a good approach or even whether the problem is one that needs solving at all. It's a union of critical, analytical, and multi-level thinking all combined and expanded. Rather than knowing what sort of thinking to apply, as I tentatively suggested previously, wisdom might be better described as turning rational thought up to 11.

And this takes us back to the limits of understanding. Wisdom here might be in recognising that these limits simply cannot be broken, that trying to probe any deeper won't result in anything useful – that we should stop when we have a definition that's actionable and have shown that further investigation won't bring any more improvements*. Likewise, I could go further with this post to better define what I mean by "best solution". But this would open up an enormous can of worms that probably wouldn't help and would make everything much longer. The wiser course of action seems to be stop here as far the definition goes. What remains is only a few clarifications and some practical consequences.

* In this particular case, we end up trying to define words by using other words, and hit the limit of what pure language can convey. I have some further musings on this which I may or may not get around to writing up eventually.

Wisdom, like intelligence, is sometimes used as a synonym for raw knowledge. If instead we say it means knowing what's the best thing to do, then clearly this requires knowledge, just as it needs both critical and analytical thinking skills. But it's not the same as any of these. We can immediately see that the correlation could be imperfect, that someone might have a huge breadth and depth of knowledge but be unable to see the relevant similarities across different fields.

This strongly suggests that wisdom is a thing that can be taught... at least, to the same extent that knowledge can be taught. The behaviour of LLMs, as per the earlier example, might offer some clues here. For these, I strongly suspect that knowledge and wisdom are strongly coupled, that all you need to make them wiser is a larger training data set and a bigger context window – they'll consider more information from more fields of expertise at more and more scales automatically*. That said, you can't really "teach" an LLM anything except by fully retraining it, which in effect gives you a whole new model. All you can really do is instruct them.

* Though of course, the quality of the output still depends on the quality of the input training data as well as the prompt.

This is not much like humans, who can certainly be taught and indeed are (mostly) capable of learning from their mistakes. Indeed, wisdom requires knowing what to avoid just as much as when to proceed. Whereas in an LLM wisdom may emerge naturally as a function of size of training data, this is (I think) likely only a trivial result of that training data containing more and more wise behaviour. It's much harder to gauge whether this happens for humans through absorbing sheer volume of knowledge, though I suspect if we're only ever told, "these things are true, learn these parrot-fashion" instead of, "here's how to evaluate knowledge", the result tends to be someone who's neither wise nor critical. Our own training certainly does matter.

Of course, LLMs don't really have beliefs and opinions in the same way that humans do. An LLM is a mass of statistical information and probabilistic weights, with no real fixed ideas at all – certainly not between conversations in the same way that humans have some ideas that they hold as almost permanently fixed. But likewise, updating our own world view in response to new information is seldom easy, just as including it in an LLM isn't as simple as telling it something in a single conversation. The analogies are interesting just as much as for their differences as their similarities.

In any case, the ability to update one's world view is not the definition of wisdom, but it does follow directly from it. The wise thinker knows when a single fact is of limited consequence and when it may necessitate a paradigm shift. They are able to judge when the new information is itself likely wrong and when it's their own existing ideas which are at fault. They consider also the metadata of who said it and why – they do not evaluate it purely on its own merits*. So the ancient Greeks were right to value to self-knowledge, as understanding one's own biases is essential in understanding how we respond to new data, but this isn't wisdom in itself, just as the ability to learn from experience is part of it, but not itself the definition of wisdom.

* The idea that ignoring the source of information is somehow actually the correct, rational approach is a curiously persistent and incredibly widespread error. 

Finally, it's obvious how the Jurassic Park scientists were intelligent but unwise. Russell Howard, by contrast, is much less intelligent but much wiser : having  a really dumb idea but realising that this would be a Bad Thing To Do. What of his grandmother ? Well, if she wakes up convinced that Hitler has returned, she's not very wise at all, but if she realises that this is so incompatible with her well-established knowledge that Hitler is long dead, then probably she's a lot wiser than her grandson. So c'mon Russel, put it to the test. For SCIENCE !

Review : Vital Organs

This book has been sat on my shelf for quite some time. A friend got it for me for Christmas and I just plain forgot about it.

I'll skip to the chase : this is the most 6/10 book that ever did 6/10. It is the very epitome of 6/10. If there was a Platonic form for 6/10, this book would be it.

It isn't bad exactly. But it isn't good either. It has enough in there to keep it interesting, but for a fairly large-print, double-spaced book with short chapters, less than 300 pages, and plenty of jokes, it's far more of a slog than it has any right to be.

I honestly don't quite know how the author did it. It's like she has the literary equivalent of poor comic timing : some of the jokes work well, but most fall flat. And all the sentences are short. Much too short. It becomes quite awkward. Longer sentences would help. It's like being out of breath the whole time. Constantly pausing is no good. It become exhausting. A real struggle to deal with. I wish she wouldn't do that.

Phew ! Seriously though, it's not a fun read. There's really no flow of the text at all, no development to anything, no structure. The author has a tendency to veer wildly and lazily into technical jargon with no or minimal explanation, the kind of attempt which I find very suspicious... it's like the author wants to say, "look, I'm not dumbing down !" and everyone else is far too polite – or scared of appearing ignorant – to ever say anything. It's not the right way to do outreach.

The main problem is that there's no obvious reason to write this book besides "collections of assorted incidents generally sell well", like an internet list that got massively out of hand. The author appears to be trying to do do main things : to give an unexpected history of body parts, and to give the reader a crash course in biology. The first part is okay – she has some interesting points about how we do more weird stuff with bits of bodies than we like to think – whereas the second is absolutely hopeless, and the combination is just as messy as any of the anecdotes. Not a single incident stood out as anything so memorable I feel the need to recount it here.

Could it have been better ? Honestly, I think not much. There just doesn't seem enough of a premise to the whole "people do strange things with bodies" to warrant a book, and no hint that there's any common underlying reason to it. If there was, things might have been more interesting. But as it is, the easy explanation hear appears to be entirely correct : because people are weird and like doing weird things. There's no pattern to any of it, nothing to generalise. It would been far better not to collect these kinds of things in a single book and leave them as isolated incidents scattered throughout the vast corpus of historical records.

There isn't much else to say about a book that I don't think needs to exist, so I'll leave this as one of the shortest post I've written in years.

The Logician's Swindle

What makes a puzzle annoying ? When is solving a problem rewarding, and when is finding out the answer just frustrating ? If we could answer this, we might get a long way towards making the world a happier place. Getting people to actually enjoy solving problems, rather than being pissed off at their opponents for discovering a flaw in their arguments, would surely benefit political discourse enormously.

I don't propose to try and answer all of this today. Instead, what I can do is address one particular aspect of the problem. I say that at least one major cause of puzzles being annoying rather than enjoyable is when you've been outright cheated, and that this happens far more often than it should.

Specifically, consider Newcomb's Paradox as described on Veritasium. The video begins :

You walk into a room, and there's a supercomputer and two boxes on the table. One box is open, and it's got $1,000 in it. There's no trick. You know it's $1,000. The other box is a mystery box, you can't see inside.

Now, the supercomputer says you can either take both boxes, that is the mystery box and the $1,000, or you can just take the mystery box.

So, what's in that mystery box?

Well, the supercomputer tells you that before you walked into the room, it made a prediction about your choice. If the supercomputer predicted you would just take the mystery box and you'd leave the $1,000 on the table, well, then it put $1 million into the mystery box. But if the supercomputer predicted that you would take both boxes, then it put nothing in the mystery box.

The supercomputer made its prediction before you knew about the problem and it has already set up the boxes. It's not trying to trick you, it's not trying to deprive you of any money. Its only goal is to make the correct prediction.

So, what do you do? Do you take both boxes or do you just take the mystery box?

Don't worry about how the supercomputer is making its prediction. Instead of a computer, you could think of it as a super intelligent alien, a cunning demon, or even a team of the world's best psychologists. It really doesn't matter who or what is making the prediction. All you need to know is that they are extremely accurate and that they made that prediction before you walked into the room.

I highlight certain parts because they feel crucial. To me, this is saying very explicitly, "don't think about this aspect of the problem, it's not important at all". Were this not so, I would otherwise object to how such a thing could be possible, and the details would certainly matter : was the machine running over a diverse sample of people, or was there something particular about them that helped its accuracy ? But no, this apparently isn't important, so whatever misgivings I have about free will and suchlike, I willingly surrender for the purpose of the test. I put them aside, still fully expecting to be fooled (I suck at logical puzzles) but in some other way.

Having made that assumption, the answer is obvious. If the machine is essentially always accurate, I take one box. It knows, magically, that this box will contain a million dollars, and I walk out happy and rich and in search for a bank offering a good exchange rate to a proper currency. 

But later in the video we get :

Here's how I think about the problem in a way that actually makes sense. You know that the supercomputer has already set up the boxes, so whatever I decide to do now, it doesn't change whether there's zero or $1 million in that mystery box, and that gives us four possible options that I've written down here.

If there is $0 in a mystery box, then I could one-box and get $0 or I could two-box and get $1,000, but there could also be $1 million in a mystery box. And in that case, I would get $1 million if I one-box or I would get $1,001,000 if I two-box. So, I'm always better off by picking both boxes.

Rubbish. Complete twaddle. You just told us that the machine is accurate and we shouldn't factor this in to our calculations, but in this way of thinking you cannot possibly ignore how the machine works. This is not even self-consistent ! By saying that the machine is essentially perfectly accurate, you've eliminated the very possibility of $1,001,000. That can only happen if the machine actually is inaccurate in some cases, which to my mind you've all but told us directly to discount.

This, then is a swindle, and one common to various logical puzzles. "Don't think about this aspect of the problem", they say, only later to say, "Hah ! You should have thought about this aspect of the problem after all, you fool !". Right, so you expect me to think you're a liar ? How is that a fair test ?

The rest of the video is a perfectly decent discussion of free will etc. (Veritasium is one of my favourite YouTube channels), but the poor description from the outset makes the whole thing a mess. Having been told that accuracy was not an issue, I expect something else I've overlooked to come into play. Naturally I overlooked determinism and all that because you told me to overlook it. The pettiness of it all annoys me quite intensely.

Don't worry, I'm not going down the free will avenue with this post. Rather, I just want to briefly outline that this kind of swindle is common to logic problems, and is itself one particular expression of a more general reason they're so often very irritating.

The closest similarity is surely the Monty Hall problem (the one with the prize goats). That one always confused the heck out of me because people never properly explained that I should have been paying crucial attention to the host's knowledge, not how many goats there are or how many doors. But any logic puzzle can suffer if you're not properly informed about what the key aspect of the problem is, or worse, if you're actively told to ignore it.

Not that framing doesn't sometimes reveal something very interesting. Wason's selection is fascinating in showing how the same people can have much more difficulty solving the same task if it's described slightly differently – especially so when the alternative form is nothing they wouldn't also be familiar with. But there, the whole point is to study psychology. No deception is employed, no swindle pulls the solution out from beneath the solver's feet. The facts are laid bare and it presents a straightforward yet surprising challenge to many people who take it. No, framing is only annoying when it's done to deliberately thwart the participant. 

There's also a common tendency for the puzzle-setter to declare the rational solution from authority, saying "this is obviously the correct solution because the alternative doesn't make sense to me". A classic example concerns people refusing small amounts of compensation when they would normally expect a much bigger payout. Time and time again we hear people declaring that accepting the small offer would be rational since they come out with a net cash gain. But to any sensible person there are a multitude of reasons why this would be an extremely foolish thing to do : accepting the initial offer may deny them any chance at the larger amount; they may simply feel insulted and disrespected, and responding to such behaviour is essentially letting the bully get away with it. It is only rational in an incredibly narrow and naive economic sense, and more broadly simply isn't rational at all*.

* Veritasium does this with a unique peculiarity, openly acknowledging that the "irrational" decision of choosing one box is the more profitable. I find this is going deep into "what's wrong with you ?" territory.

Again, this is a sort of swindle, denying the opposing argument by forbidding debate rather than engaging with it on an equal footing. You thought things were going to be fair and above-board only to find out that they were anything but, that the answer had already been decided without you.

Another similarity is the pettiness. Veritasium didn't have to pull the rug out from under the viewer's feet any more than anyone has to accept that getting a smaller payout is somehow rational. 

Very occasionally, I've run public surveys to help me with my own research. I've tried to ensure the wording was extremely careful, including omitting details when this would bias the result. For example I once ran a public poll on how many groups of points – galaxies – people could see in a plot, deliberately not telling them what they were looking at. Some people objected that there wasn't enough information (e.g. what sort of scales they should be considering), and I sympathise that they might find this annoying. But for me this was the whole point, to gauge what people's natural reactions were : I wanted to know if they would instinctively identify the same groupings that appeared natural and obvious to me (most of them did, as it turned out). I needed to know if my additional knowledge was biasing me, or if the groupings I identified would be readily visible without this extra information. 

The point here is that there's absolutely no reason for misdirection. It's perfectly possible to account for this in a way that will give you a meaningful result to the question you're asking. Sometimes, this can only become apparent after the fact, but in those cases the participant should feel relieved, not annoyed. Annoyance only happens when the misdirection was unnecessary. 

A second personal example : group meetings back in my PhD days. These served the valuable purpose of getting the students used to dealing with tough questions. But they also turned the experience into a weekly grilling that made the whole thing quite intensely annoying... instead of having an enjoyable, low-stakes discussion about science, we had to deal with supervisors being deliberately over-critical. That we all knew full well what was going on didn't help in the least. It would have been fine if such sessions had been clearly demarcated and set aside as such, with regular meetings more about science for its own sake. Trying to pretend this was how scientific discussion should happen, though, was just unfair.

Again, there was no reason for the misdirection. This too was a sort of swindle. Oh, you think you're here to discuss your work ? You thought I was being harsh because I wanted to be ? Hah hah, fooled you ! The idea that maybe they could have just not done that was never raised.

On an grander scale, problems with the alternatives to dark matter. This too feels like something of a swindle : proponents often raise objections to dark matter which are based entirely on the properties of the ordinary matter we can see. They make highly dubious inferences about the necessary connection to the dark matter they're trying to demonstrate doesn't exist, saying that the lack of a naively-expected correlation proves it can't exist. Some of these problems can become obvious, but sometimes it's worth spelling this out at the high level because it's all too easy to lose sight of the forest for the trees. Once you start questioning the underlying assumption and realising that maybe the connection isn't so direct after all, often the whole thing falls apart.

And in other arenas too we find possible swindles. As I've covered before, thought experiments become extremely annoying when changing a small detail would profoundly alter the result but the instigator refuses to consider any variation : no, you must focus on this aspect of the problem because I said so, even if my scenario is actually bunk. Just like insisting someone should accept a miniscule payout, it's disrespectful not to think the other person's opinion might have some value. 

Likewise with analogies. An indirect analogy can be extremely powerful when the relevant aspect is sufficiently similar to its comparison subject, becoming thought-provoking in both its similarities and in its minor, extraneous differences. When an analogy is intended to be direct, though, the seemingly-extraneous details can become crucial, so expecting people to shut up and ignore them is not realistic. It's extremely difficult to focus on the "relevant" bit (usually declared by authority) when there are obvious deficiencies in the whole thing. Conversely, it does no good to pretend similarities don't exist when they do, or to overlook them on grounds which are actually minor details or only quantitative differences.

All this sets out some conditions for when puzzles becoming annoying, and gives us a rough working definition : The Logician's Swindle is the use of unnecessary misdirection from a position of unjustified authority.

This is similar to but not quite the same as the Magician's Choice. In the latter, we know we're being denied crucial information, misdirected, and otherwise deceived. We go in with eyes open knowing we'll almost certainly be tricked and often paying for the privilege of suspension of disbelief. We know we won't be able to solve the problem and we enjoy our failed attempts to work out what's going on.

The Logician's Swindle is altogether nastier. Here, we're supposed to have all the information we need to reach the "correct" conclusion, but we find only afterwards that actually we don't – with the swindler often denying this for the sake of making us look foolish. And the conclusion itself may be open to dispute but the proponent argues from a completely artificial authority that it isn't. Worst of all is that "mistakes" can (though do not always actually) carry real-world consequences. In short, it's a scam : a discussion that should be in good faith which actually isn't.

And that's why I hate logic puzzles.

I'd Like To Teach Machines To Think

Today, a couple of contrarian pieces claiming that maybe LLMs do think and reason after all.

That is, not in a namby-pamby, "it's just something similar enough to thinking that we might as well call it that" sort of way. This is perfectly reasonable. I stand by that myself. To get hung up on saying "they're not really thinking" every time someone casually uses this instead of "processing data" is frankly just annoying, not productive. Likewise for intelligence : if they're taking input data and producing coherent output, well I call that a form of intelligence at the very least.

No such linguistic sleights of hand are to be found here though. No no, these pieces are much closer to the dreaded C-word... consciousness.

The first article makes much the weaker claims of the two. This one touches on the self-awareness issue, but its main point is simply that they're doing something more than pure word prediction.

Modern LLMs (Claude, GPT-4, and others) have an interesting feature, the humble thinking/reasoning tokens. Before generating a response, the model can generate intermediate tokens that the user never sees (optional). These tokens aren't part of the answer. They exist between the prompt and the response, modifying the context that the final answer is generated from and associated via the attention mechanism. A final better output is then generated.

Every token between the prompt and the response is, in information-theory terms, an opportunity for drift. The prompt signal should attenuate with distance. Adding hundreds of intermediate tokens into the context should make the answer worse, not better. But reasoning tokens do the opposite. They add additional machine generated context and the answer improves. The signal gets stronger through a process that logically should weaken it.

This process improves performance. That thought should give you pause, just like how a thinking model pauses to consider hard problems before answering. That fact should stop you cold.

As someone who's not in any way, to any degree, expert in LLMs, this is highly unintuitive. After all, when I start thinking deeply, I don't expect that the more I think, the more off-track I'll go and the worse my output will become. I expect that the more things I take into consideration, the more deeply I delve into the problem, the more accurate and insightful will be my final output. I guess I presume LLMs are doing something broadly analogous in that they are (in effect) considering ever-more voluminous amounts of text in producing their final response after a chain-of-thought. The more they consider the relations between different words, the better should be their final output.

I asked ChatGPT about this, and its (considerably shortened) response was as follows :

I think the article is smuggling in a dodgy intuition there. The right baseline is not “extra tokens should obviously make things worse”. In a transformer, later tokens are not forced to overwrite earlier ones in a simple serial way. Attention lets the model keep referring back to whatever parts of the prompt or scratch work are useful, so intermediate text can genuinely help by turning one hard prediction into several easier ones. That is exactly why scratchpad methods and chain-of-thought prompting often improve performance on multi-step tasks.

So I would rephrase the issue like this : intermediate steps should not be expected to degrade performance in general. They should be expected to help when they function as a good scratchpad, and to hurt when they function as self-generated noise. The article seems to assume that the mere presence of extra tokens ought to be harmful. That is too simple. The real trade-off is not between “direct answer” and “more text”, but between “useful decomposition” and “error propagation”. One small clue that this trade-off is real is that CoT can also make models more confident when they are wrong, which is exactly what you would expect if self-generated reasoning sometimes stabilises mistakes instead of correcting them.

This seems not crazy. It then seems too much of a stretch in the article, to me, to claim that because the model is reasoning "in the context of a probability distribution", it's still doing something directly (and I emphasise the emphasis here most emphatically) analogous to some aspects of human reasoning. I think we have the capacity for a much deeper, truer understanding than any LLM has or ever will have.

If you wish to reduce this to "just" token prediction, then your "just" has to carry the weight of a system that monitors itself, evaluates its own sufficiency for a posed problem, decides when to intervene, generates targeted modifications to its own operating context, and produces objectively improved outcomes. That "just" isn't explaining anything anymore. It's refusing to engage with what the system is observably doing by utilizing a thought terminating cliche in place of observation.

None of this requires an LLM to have consciousness. However, it does require an artificial neural network to be engaging in processes that clearly resemble how meta-cognitive awareness works in the human mind. At what point does "this person is engaged in silly anthropomorphism" turn into "this other person is using anthropocentrism to dismiss what is happening in front of them"?

This doesn't feel warranted to me. For sure, humans probably use linguistic heuristics in place of "actual" reasoning more than we like to, err, think. LLMs manipulating text, in ways not matter how arbitrarily complex, does not indicate any evidence of actual, true reasoning and thought. It's just an incredibly clever way to predict words, and I see no evidence of anything deeper going on at all. No sentience, no awareness, no emotions, no preferences, no inner light. The LLM literally does not exist when it isn't prompted. It has no consciousness, no subconsciousness, no self of any kind.

The irony if course delectable... I prefer the LLM's claim that it isn't thinking to the human's assertion that it is !

Before I go to the next article, I also have to mention a recent discussion with a very interesting claim indeed :

The thing is, Yann LeCun was actually right. Purely text-based LLMs never learned that if you push a table, you also move the objects on the table. What happened instead is that LLMs became “multi-modal” and made to accept images, audio, and video as input as well as text. So “AI” did learn that if you push a table, you also move the objects on the table, but Yann LeCun was right that they did not learn it purely from text.

Despite this coming from a genuinely proper expert, I struggle to believe it. It simply doesn't match my experience with LLMs at all... well, maybe a little bit with GPT-3.5, but even with that hilarious dumbass, only a little bit. Causal connections between objects don't seem particularly difficult to establish via pure text : if you move a table, you move all objects on that table... and LLMs are surely very good at knowing what word represents a literal object. This really doesn't feel like something that should present any difficulty. 

I wish 3.5 were still with us... I'd love to test it.

GPT-5.4 says that this claim isn't correct, but that there is an interesting point behind it. That is, multi-modal models do help LLMs learn things that humans never bother writing down, but that "text-only models clearly do learn a fair amount of everyday physical regularity from language". This I would definitely believe. Without some very specific documentation though, the claim they can't learn something as basic as objects being moved when their supporting table moves is something I'd be very reluctant to concede. The current "car wash" problem is an interesting reminder that LLMs aren't human, but not, I think, proof categorical that they're totally lacking in any sort of reasoning capacity whatseover.

On to the second, much more full-throated article.

The fundamental case against the “I” in AI is that intelligence is organic, derived from sensory interaction with a physical environment. Agüera y Arcas turns the tables with the premise that computation is the substrate for intelligence in all life forms. The claim builds on an apparently crude proposition: prediction is the fundamental principle behind intelligence and “may be the whole story”.

I react quite instinctively against this, essentially with, "fascinating, tell me more about your stupid idea !". That is, there are some things I think are gloriously weird. I love their sheer audacity, may or may not hold them respectable, but don't believe them for a microsecond. I do not mean "stupid" here in an especially pejorative sense, but if you can't already understand it, then I probably can't explain it.

A central tenet of What is Intelligence? is that every form of life is an aggregation of cooperative parts. Links proliferate through patterns that enable increasingly complex functions. When Agüera y Arcas says the brain is computational, it’s not a metaphor: it is not that brains are like computers, they are computers.

He is erasing a familiar conceptual boundary here: intelligence does not prompt function, it is function. Intelligence, he argues, is a property of systems rather than beings, and function is its primary indicator. A rock does not function, but a kidney does. This is demonstrated simply by cutting them in half. The rock becomes two rocks, but the kidney is no longer a kidney.

So does a kidney have intelligence? Or an amoeba? Or a leaf? These questions are opened up, along with the question of whether Large Language Models have intelligence, which may a better way to frame it than asking whether they are intelligent.

In another discussion, I could not make myself understood when I tried to say I think that awareness is something one has, not what one is. This is extraordinarily hard to explain if you don't already "get it", but cannot for the life of me understand the claim that experiences are literally the same as physical brain states. To me this is an absolute non sequitur, with the evidence being possibly the clearest that could ever be presented for anything ever. I won't try and do it again – this blog is chock-full of that kind of stuff as it is – but maybe it's still useful to frame how I think about LLMs.

I do not have any issue with humans eventually constructing some sort of true AI. I think it's perfectly possible we could construct a chip or something which would give rise to (or otherwise access) consciousness. I do not think an LLM will ever do that, because it's literally just rearranging text. I see no more reason an LLM could be conscious than the text of a newspaper if it were cut to pieces and thrown into a whirlwind. This is why that when I say it makes good sense to describe LLMs as intelligent and reasoning, I do so with a quite monumental proviso that this in no way whatsofuckingever implies they do so much like humans – at least, not with regard to the full scope of how humans think.

 Maybe some bullet points would help ? Well, let's try. I claim :

• LLMs can be said to think and reason in that they process input data to produce sensible outputs. In certain domains, they can already do so with an accuracy that rivals and exceeds humans.
• LLMs function as more than pure word predictors; they can generalise and abstract to a useful degree, though highly imperfect and not on a par with humans except for the stupid ones (of which unfortunately we must contend with many). They do some things with appreciable, meaningful similarity to humans.
• LLMs are nevertheless just mechanical. They don't have an inner life, are incapable of feeling anything, have no desires, no sensations, and no awareness of even the smallest degree. You can't program a true AI, you have to build one. BUT...
• ...who on Earth would want a mechanical mind that would potentially suffer or try and eat us or whatnot ? Pointless. Far better to stick with an LLM-like route and go for pure tool development; if you want to bring new souls into the world, there's exactly one way to do that, and it ain't about building robots in your garden shed.

There, does that clear things up ? I doubt it. This has been the most decoherent of Decoherency posts, so if you think I'm being self-contradictory, you're wrong and I don't care. But I feel slightly better, anyway.

Everything Is Too Easy

Today, a short look at a BBC article claiming that the modern world is making everything too easy. Yes, really.

It is, of course, all about the age-old argument as to whether making things easier makes us stupider. There's a fair point to be made for "use it for lose it", but the question should really be about when and whether skill atrophy actually matters. There's no point worrying that your driving abilities will degrade if you switch entirely to public transport, and so long as you can guarantee your new option is reliable and available, there seems no point at all in bemoaning this (I've covered this issue before, of course). Indeed, the switch should be the whole goal. That's how progress is supposed to work, or it wouldn't be progress at all.

Likewise, the ancient Greeks believed that warm showers would make them weak and effeminate, whereas most modern people would say that cold showers just make them miserable rather than toughening them up. There are more than enough other issues to deal with without also having to suffer needlessly. I believe we generally do better when our basic needs are met rather than having to struggle for them : I prefer to live in Star Trek's comfortable, scientific Federation rather than with Dune's tough, religiously fanatical Fremen. Struggle should be reserved for the things you want, not the things you need.

Still, there is a case to be made that it would be a Bad Thing if your overall intelligence started decreasing because you stopped trying to think for yourself. You do need to actually engage your brain, and all skills are maintained by continuously operating near their peak levels. What I struggle to see is how it's ever possible (except maybe if you're far too rich) to make your life such utterly comfortable, so devoid of any challenge whatsoever, that this ever becomes of any actual concern to anyone at all.

While modern technology can streamline day-to-day life, making everything from dating to food delivery more efficient, it may come at a cost: early data suggests that our attention span may be shortening, critical thinking capabilities weakening, emotional intelligence fading, and spatial memory getting worse as we offload human tasks to our devices. The technological optimisation doesn't seem to be making us happier, either: despite the continual digital assists and enhanced communication of social networks, people still report high levels of stress and loneliness.

Yeah, but that's a lot of social media ends up with interactions with people we'd never want to engage with in real life. You then have to spend ages dealing with stupid people under the dubious grounds that you'll otherwise end up in an echo chamber*. This is partly because most people are, in fact, very stupid, and partly due to algorithmic rage-bait manipulation.

* This apparently being something unique to social media : of course, you can walk away from morons in real life, but online we're expected to deal with them calmly and rationally and take them very seriously for some reason.

That's why a growing number of people are restoring to the hottest new trend: "friction-maxxing", or rebuilding tolerance for inconveniences. The idea is to find tasks or ways of doing things involve a level of difficulty, time or patience. This could, for example, involve going "old school" and swapping digital tech tools for analogue solutions, such as reading rather than watching YouTube, navigating by road signs in place of Google Maps or calling a friend for advice instead of consulting ChatGPT.

But our brains operate on a "use it or lose it" principle, says Mark. Experiments in animal models show that effortful learning keeps new neurons in the brain alive. Studies also show that cognitively-stimulating activities like learning an instrument, reading, playing games and doing puzzles can preserve cognitive function as we age.

I don't get it. Who is finding their life so convenient that they miss the difficulties ? More importantly, HOW CAN I BECOME ONE OF THESE PEOPLE ? I find that one incomprehensibly weird. How are you not just exchanging one set of difficulties for another ? For example, if I no longer have to worry about how to write the code, I simply have more time for thinking about the problem I wanted to solve in the first place. I can't imagine ever reaching the end of the chain, as it were. Yes, it's good to have some amount of very low-level technical knowledge (some degree of grit for the mill, I guess), but I can't imagine who's living in such a utopia that they've already reached a state of post-scarcity difficulties. I'm a lot happier, not stupider, if I don't have to deal with segmentation faults.

According to one band of experts, the features of our digitised existence – constant notifications, 24-hour news and endless social feeds – can hijack this attention system, resulting in cognitive overload, mental fatigue and trouble focusing.

Much like the research on the effects of technology on our mental capacities, the studies of digital detoxes show mixed results. Some breaks from technology lead to better mood, improved focus, lower stress and more social connectedness, while others show the opposite or null effects. One 2014 study found that restricting screen time at a five-day nature camp improved preteens' emotional and social intelligence, while another 2019 study of university students found an increase in loneliness after abstaining from social media for one week.

Now that one makes a lot more sense. I can definitely see problems with "everything devices", like phones : I have a tendency to stockpile stuff for later consumption rather than immediately reading what interests me; I definitely fall into the infinite scrolling trap.

As per a quite different recent post, this is not necessarily a bad thing. Indeed, to a degree I even welcome distractions of my own choosing. I find it helps keep my focus maintained for longer overall if I split it up into chunks, every so often checking something very simple (like a news feed or social media) that doesn't derail my train of thought. Usually, this sort of back-and-forth doesn't become a problem when working, but I'm as guilty as anyone of scrolling on my phone rather than actually watching TV. Especially if I'm watching a show on my computer... the temptation to just quickly check the feed again is, for whatever reason, quite hard to resist. 

But to me this not a technological issue, it's a design choice. I love my digital notebook because it's total lack of features mean I use it for writing notes and reading long web articles, and that's it. Single purpose items, be they digital or otherwise, innately demand more attention because you can't just flick a button for more content. Moreover, so long as I put my phone away, I'm not tempted to go and check it. A single purpose device doesn't feel boring : once you're given something to do, and nothing to distract you from it, you'll just do that thing instead of trying to do dozens of things at once. Printed books do this, but so do well-designed digital products.

But even if friction-maxxing isn't the end-all solution we've been waiting for, "it doesn't hurt", says Mark. "If people are putting in effort, it makes them more intentional and thoughtful." Analogue hobbies such as crafting, gardening or reading – which involve friction as opposed to scrolling or streaming – can act as "active meditation", calming the mind and reducing stress. One 2024 study of more than 7,000 adults living in England found that those who engaged in crafting or the creative arts were more likely to report significantly higher life satisfaction, a greater sense that life is worthwhile and increased happiness. 

"I realised that a good life isn't an easy life," Semple says. "There's an enjoyment that you're cheated out of when you take the easy route."

And this is fair. Most analogue activities are innately more focused because there are no equivalents of "everything devices" or constant notifications. Multi-functionality has its place, but it's not easy to stay to track with such things. The real difficulty is perhaps the web itself. It doesn't make any sense to have a separate physical device for every website or digital tool : some purposes, like photography, can easily and sensibly separated, but many others simply can't. 

What's the solution ? Self discipline is part of it, but it's hardly the whole answer. On a totally different video, I liked the point that it's simply not fair to expect people to be able to resist something that's designed to capture their whole attention. So better designs are also needed, e.g. alternatives to infinite scrolling, more nuanced control of feeds, easy options to control which notifications come through and when (Gmail and DuoLingo, shut the fuck up already)... I don't think this would actually be difficult to do and it would cost exactly nothing.

The problem is getting corporations to see that making us engaged with more and more things but each for less and less time is, in the long term, a stupid metric to gauge product success, and that providing us with one good service is better than the option of a hundred crappy ones. How we reign in this tendency, I don't know.

Listening To The Voices In Other People's Heads

Here's a very nice long read from the Guardian about trying to understand what's really going on inside our heads.

Recently I concluded that any attempt to understand what consciousness actually is is likely hopeless. Trying to understand what we mean by experience when literally all we have access to is experience is inevitably circular. But the effort itself isn't fruitless. We can all of us have different preferences for what we think is going on – non-physical, spiritual, purely materialistic – and that discussion is often productive, if only in understanding how people reach radically different conclusions from the same data. More promisingly, it gives us a better handle on how we go about defining things, how we grapple with the imperfections of mapping language onto reality.

But there were also two more directly productive outcomes described in the Aeon essay I looked at last time. One was that we could understand in some detail the neural correlates of consciousness, the processes occurring within the brain that are associated with what we think and experience. The second, somewhat subtler issue, was that we can still take a reductive approach to different aspects of consciousness : we can describe it in terms of different levels and content, and in so doing get back to something we can discuss in familiar scientific terms. Just as we don't have to worry about what a quark is really made of to understand how it behaves, so too we can tackle the subject of minds.

The Guardian piece is complementary to the Aeon article in that it leans more heavily in this direction. As it begins :

A neuroscientific perspective on consciousness might tell us something about its neural correlates, but it is unlikely to tell us much, if anything, about the nature of thoughts or the textures of inner experience; it’s the wrong tool for that job. So what might we learn about consciousness if we gave more weight to the view from inside the experience – the phenomenological viewpoint ?

For example, it describes William James' comments on something I've found extremely strange for many years :

“Suppose we try to recall a forgotten name,” he writes. “The state of our consciousness is peculiar. There is a gap therein; but no mere gap. It is a gap that is intensely active.” A sort of ghost of the absent name haunts the empty space in our consciousness, he suggests, making us “tingle with the sense of our closeness, and then letting us sink back without the longed-​for term”.

He goes on: let someone propose a candidate for the missing name, he suggests, and even though we have no consciousness of what the name is, we are somehow conscious of what it is not, and so summarily reject it. How strange! Our consciousness of one absence is completely different from our consciousness of another. But, he asks, “how can the two consciousnesses be different when the terms which might make them different are not there ?” 

The feeling of an absence in our minds is nothing like the absence of a feeling; to the contrary, this is an absence that is highly specific and intensely felt. Thoughts glimpsed from some height of awareness but somehow not yet formed, much less put into words or images – this is the subtle terrain James invites us to explore with him.

I very much agree with this. It's always seemed weird to me how we can think in complete sentences. We – or rather I, and I'll get back to why the distinction matters later on – don't really sense the words falling into place, they just come out like that. Clearly they must have been assembled at some point, but somehow this happens without us knowing about it ! 

Perhaps even weirder is that sensation when grappling with a complex problem : at the point of reaching a possible solution comes a very distinct sensation of raw, unstructured thought, some quasi-awareness that the answer has been reached but without being able to articulate it. That moment is crucial. If interrupted in this momentary phase, the proto-thought may be lost entirely. But if it's seen through to completion, the thought crystallises into language : something we can easily memorise, recall, and communicate with others.

Much of the article is then concerned with whether we can access this much lower level of thinking in some way. If we want to understand consciousness, can be go beyond language ? Maybe that's too ambitious. To start with, can we at least access thoughts at the stage that they become coherent ? 

Step forth, Russell T. Hurlburt :

For half a century now, Hurlburt has been scrupulously collecting reports of people’s inner experiences at random moments – and just as scrupulously resisting the urge to draw premature conclusions. A die-​hard empiricist, he is as devoted to data as he is allergic to theories... I’ve been going around with a beeper wired to an earpiece that sends a sudden sharp note into my left ear at random times of the day. This is my cue to recall and jot down whatever was going on in my head immediately before I registered the beep. The idea is to capture a snapshot of the contents of consciousness at a specific moment in time by dipping a ladle into the onrushing stream.

What he is after in his research is the “pristine inner experience”, by which he means a sample of human thought “unspoiled by the act of observation or reflection”. Like James, Hurlburt acknowledges that the act of recalling and describing an experience is bound to alter it, but he believes that his method can get us closer to the uncontaminated ideal than any other.

In some ways this should be extremely easy. Again, we definitely do have coherent, structured, linguistic thoughts, and writing those down is straightforward enough. Indeed, as this fascinating BBC article describes, it's even possible to read these directly from the brain. This doesn't require participants to mentally try and speak : to a degree, thoughts can now be extracted at a lower level than this.

But then again, even the most fully-developed thoughts can be extremely slippery. As I've noted, once you put pen to paper you engage in a sort of self-conversation, thoughts and beliefs becoming highly flexible once you start reflecting back on them from an external input. Take a thought from in here and put it out there and it inevitably changes, even if only just a little. 

Still, this can largely be avoided : once a sentence is formed, it can be written down. A more difficult aspect of the problem is trying to disentangle that bit of coherency from everything else we're thinking about :

I took out the little pad provided by Hurlburt and jotted down this thought: “Deciding whether or not to buy a roll.” I know, not terribly exciting, but it seems very few of my mental contents are. I was thinking ahead to lunch and wordlessly deliberating whether to buy a fresh roll for a sandwich or do the responsible thing and use up the heel of bread I had at home. I was also conscious of the pattern of the skirt – an unflatteringly large plaid – worn by the woman standing in line ahead of me. 

Was that observation part of the moment in question, or did it come immediately before or after? I couldn’t say for sure. (How long does a moment in consciousness last ?) And what about the pervasive smells of freshly baked goods and cheese ? These both preceded and followed the moment under examination, but were they present to my awareness at the beep ?

Throw in just a few complications and suddenly the problem becomes much more difficult, maybe even impossible. Words ? We can attend to them. The whole facet of experience, including our different senses, how they affect us, when they occur, when we assemble a sentence ? That's much more fuzzy. Trying to describe anything before the point of coherency may even be a non sequitur. Maybe we can pin down a bit more about the process by determining what we're currently experiencing (e.g. which senses are given priority over the others, where and when we give out conscious attention to language rather than sensory experience, what counts as thought, etc.) but there will be some limits as to how far we can get with this.

Some really fascinating things have come out of the research though :

The first finding, to which I can personally attest, is just how little most of us know about the characteristics of our own inner experiences. “That’s probably the most important finding that I’ve got,” Hurlburt said.

Important, yes, but I think the other findings are a lot more interesting : 

Inner speech, which many of us – including many philosophers and neuroscientists – believe is the common currency of consciousness, may actually not be all that common. Hurlburt estimates that only a minority of us are “inner speakers”. So why do we think we talk to ourselves all the time? Perhaps because we have little choice but to resort to language when asked to express what we are thinking. As a result, we’re “likely to assume that’s the medium for inner thought”.

But that doesn’t make it true for everyone. Fewer than a quarter of the samples that Hurlburt has gathered report experiences of inner speech. A slightly lower percentage report either inner seeing, feeling, or sensory awareness. Still another fifth of his samples report experiences of “unsymbolised” thought – complete thoughts made up of neither words nor images. Hurlburt has suggested that we fail to recognise the diversity of thinking styles because we lump them all together under that single word – thinking – and assume we mean the same thing by it, though in actuality we don’t.

Aphantasia and the lack of inner speech is something I've covered many times before, but this is something beyond that. It's something I know I must have but am absolutely incapable of imagining. A truly pure thought consisting of... what, exactly ? Not language. Not any of the senses. Just pure electrochemistry, I guess. That's absolutely wild. 

EDIT : To a degree, I can imagine this. My inner monologue is pretty incessant, but it's not constant. There are times when it shuts up and all I have is sensory experience and emotions. What I absolutely cannot do is properly articulate what's going on. Is there still some processing going on using language at the lower levels, ready to be raised to my awareness for perusal when required ? Or is it fundamentally different at the earlier stages and only converted to language later on ? More below.

I wonder if the brain scans described in the BBC would be capable of interpreting these in the same way as for the (to me normal) case of thinking with an inner voice or eye. Perhaps it's like blindsight. That is, maybe our brains are all doing basically the same low-level stuff, but sometimes not everything is raised to whatever part it is that brings it to conscious awareness. Or maybe, even more interestingly, we don't all work in quite the same way. Regardless, scans of people who aren't thinking with inner speech, imagery, or any kind of structured thoughts would surely make for a fascinating comparison. Would the scan reveal the same thing as in those with well-defined internal monologues or would it show something else altogether ?

Another researcher suggests a different and more holistic approach :

The field’s focus on conscious perception has led it to overlook the 30-50% of mental experience that is fed to us by our minds rather than our senses, Kalina Christoff Hadjiilieva contends. “Consciousness is just one function of the mind,” Hadjiilieva told me during one of a half-​dozen interviews, this session over a cup of tea in my garden. “To focus on conscious thoughts is like focusing on the leaves of a tree and trying to understand them in isolation,” she said. “The tree is the mind, and there’s a lot more to the mind than consciousness.”

The degree to which the mind wanders appears to be surprisingly important :

Hadjiilieva conducted an experiment with long-​term meditators (mindfulness practitioners). These are people who have been trained to still their minds but also to notice the precise moment when that stillness is broken by an errant thought, which Hadjiilieva found happens every 10 to 20 seconds or so even in these trained minds. (“The big lesson of meditation,” she said, “is that the mind cannot be controlled.”)

This makes intuitive sense to me, and again maybe reveal something about the structure of thought processes. The way I like to work even, when in a state of relative focus, is often to flit back and forth between a couple of different things at once. I like to check my emails and glance at the news quite frequently, only going into really deep focus every once in a while*. WIth some tasks this works very well : it's like I have my brain keep working on the other thing in the background while giving my consciousness time to rest by attending to something easier. What's crucial for me, though, is that these must be activities of my own choosing. Being disturbed by an external influence is a big no-no, If someone interrupts me then the process is instantly broken.

* Though my work habits vary considerably. For code I nearly always concentrate on the code and absolutely nothing else, with a similar situation for most difficult problems. It's for the routine, less cognitively demanding tasks that I prefer to have multiple tabs open, as it were.

This all ties is quite nicely to the earlier discussion :

Hadjiilieva and her colleagues noted a jump in activity within the hippocampus, a key component of the default mode network that is involved in not only memory but also learning and spatial navigation. To their surprise, the leap in hippocampal activity preceded the arrival of the thought in the meditator’s consciousness by nearly four seconds – an epoch in brain time, and far longer than it takes for a sensory impression to cross the threshold of our awareness.

You might wonder if this further shifts my uncertainty about the apparent non-physical nature consciousness. In this case, it doesn't. I've already covered similar experiments regarding free will, and here it seems to me that no neural correlate could be anything remotely like subjective experience : how do some electrons whizzing about resemble the smell of a daffodil or the feeling of anger ? They simply don't, and to assume otherwise is to completely miss the point of the Hard Problem. But to build on from the previous post, this is still very interesting stuff : 

“Something is going on prior to awareness,” Christoff Hadjiilieva said, but she’s not sure exactly what it is or why it takes so long. This finding indicates that a spontaneous thought must undergo some sort of complicated unconscious processing before finding (or forcing) its way into the stream of consciousness. For Hadjiilieva, the mystery she’s uncovered points to what she regards as the “really hard problem of consciousness” – how the contents of the unconscious form into thoughts that sometimes find their way into our awareness, and sometimes don’t.

Well, that's definitely a hard problem, and maybe it would even be better to call it the hard problem. The Hard Problem of the philosophical sense may well turn out to be the Impossible Problem : we literally can't understand our own subjective experience, since by definition this is all we have access to. In that case some relabelling makes good sense. It seems very reasonable that the time delay points towards the brain doing some unconscious information processing before raising it to our awareness, and understanding how and why this happens seems extremely difficult but far from outright impossible.

The article concludes, in typical Guardian fashion, with a warning of the dangers of capitalism in preventing our minds from going about their productive, unguided wanderings, as well as the difficulties of persuading people to treat research into the subconscious as serious science. Perhaps the author should have read that BBC story. If you can access this with a machine, the danger may not be that nobody takes it seriously, but the exact opposite.

The Truth About Utility

What makes a useful definition ? Originally I had a much more philosophically pretentious post semi-drafted for this, and I may still do that one separately. But various recent discussions have taken me down a very different path, one which might be more, err... useful. So let's start with this one and see if I ever get back to the nature-of-reality version in a future post. 

A good definition, I think, must surely be something which is widely applicable but also specific : things which happen frequently but not always. It should be readily distinguished from similar counter-examples. Crucially, it cannot be something which either never happens or always happens. It shouldn't be something which forbids itself entirely or makes it inevitable or ubiquitous. It should describe a specific thing that actually sometimes happens, or is at least conceptually valid and distinct from other, similar terms. 

What I see people doing is trying to make things true or false by setting their definition up in such a way that it cannot ever fail, and that to me seems like a mistake. This doesn't mean we can't have productive discussions, but it does, I think, impose some extremely unhelpful limitations. 

Let's do this one by example. First I'll look at cases where people define things such that they can't ever happen, and then the reverse, related case of defining them such that they're inevitable. Both in my view are counterproductive mistakes. They are terminology problems but they prevent us from getting at what we really mean, which is usually much more interesting.

1) Defining things out of existence

If we define something with such precision, such high standards, or such that it involves a logical contradiction that it can't ever be true, then I submit that this isn't a useful definition at all. Furthermore, it's likely not what we really mean when we use the term in everyday discourse.

Malevolence : Plato and other ancient philosophers help that nobody would knowingly do evil. I forget who it was who described it explicitly (possibly several people), but the basic idea was that if you knew something was wrong, you couldn't possibly do it. You might still carry out an immoral action, but you'd be misjudging, thinking that the gratification you would get would outweigh any negative consequences there might be for anyone else. Alternatively, you might do so only because you hadn't realised the existence, extent, or nature of those negative consequences.

I think this is a deeply mistaken view of humanity. As per the link, people certainly carry out heinous acts in full knowledge of their full consequences, sometimes this being the very reason for their behaviour rather than a side-effect. Or they may know but simply not care. But they aren't, I think  carrying out a mental calculus of where the balance lies. Even if they were, this would still make the word – or notion of wilful harm – meaningless. The point for most discussion is that people cause each other harm sometimes because they want this to happen, not out of ignorance. Anything more beyond this rapidly leads into such convoluted nuances that the definition collapses into uselessness. 

Or to put it another way : "Sure, he committed the murder, but it wasn't out of malice : that's impossible, so he must've done it because he mistakenly thought his pleasure at the victim's suffering would outweigh their actual suffering". 

To me this makes the word unproductively useless, trying to define the thing out of existence. Surely that points towards this meaning not being what we truly meant : the important thing is that people inflict harm on others for its own sake, and inferring anything further is best avoided altogether.

Altruism : In the opposite case, my partner likes to say that nobody is really altruistic. Everyone acts, she says, because they believe there will be some benefit to themselves, even if that reward is purely emotional. In the extreme case, someone might be give their own lives to save others, not because they thought the value of the lives they saved outweighs their own, but because of the fleeting emotional reward they themselves will get from knowing the others will live.

This too I think is surely putting more on the word than it can bear. The point of altruism, I'd say, is that we sometimes value others more than ourselves and act to bring a net benefit to them even at the expense of our own status. Start demanding that we get no emotional reward at all and again the term has been defined out of meaningful existence. This makes it utterly useless, and surely, therefore, this can't be what most of us mean most of the time when we use it. I'll qualify this a bit more later, but that general-case point is the one I want to focus on.

Knowledge and understanding : I've covered the nature of LLM-outputs several times, most recently here but also e.g. here and (tangentially) here. More on those more directly in a minute, but a closely related claim is whether they can be said to truly understand anything. I think they can, in the carefully qualified sense that a) they have access to some form of information; b) they form connections between different pieces of information; c) they act in a logical, coherent way to predict how things behave in novel situations. Not perfectly, it's true, but more interesting by far is that they do it at all. 

Now for sure, this is not the same sort of understanding that humans have. But its qualitative similarities, in my view, outweigh and are far more interesting than the quantitative differences between silicon and neural understanding. I think it's just not at all useful to say that "meaning only comes from humans ascribing this to the output". This is so inevitably necessary that it adds nothing useful to the discussion : well who else was going to be reading the output then ? And if you define "understanding" to be only a human thing, then it's tautologous that no non-human will ever have it. That's cheating.

2) Defining things into existence

We can now see how the reverse is also true : if we define a thing as being completely unavoidable, we won't get anywhere.

Hallucinations : See the links in the previous entry as this follows directly from the previous definition. I did initially agree that it was sensible to describe all LLM output as a hallucination, but I changed my mind some time ago. Given that they are now able to process complex (and multi-modal) information in a way that closely aligns with human expectations, and can in fact exceed our own predictive capacities at least some of the time, I now no longer think describing their output at purely hallucinatory makes much sense. 

It's more useful, I think, to say they're hallucinating (in their own peculiar way) when their output has no connection whatever to the input data or prompt. This is much more analogous to human hallucinations in which we see things which aren't there. I would still agree, provisionally, that LLMs treat all information has having a much more similar level of validity than humans do, and undeniably they have some qualitative as well as quantitative differences from human thought. But they are very clearly not purely fabricating stuff all the time : more often than not, they're processing their inputs quite sensibly.

Importantly, the claim that all LLM output is a hallucination is consistent with the notion that they don't understand anything. I'm not claiming incoherency here : I'm claiming that these definitions should be discarded because they aren't useful, not because there's any inherent problem as such. The alternative definitions I've suggested are, I think, better only because they are more flexible and specific, allowing us to describe things in more detail, not because they eliminate any inconsistency.

God : Don't say I'm not ambitious ! The old argument that god is necessarily perfect and perfection necessarily exists... well, surely this is the ultimate case of trying to assert truth by definition. God is a perfect what, exactly ? A perfect square ? A perfect teapot ? Well, if a perfect teapot exists, where is it ? Could it be Russel's Teapot, somewhere beyond Earth's orbit ? Surely not, because if it was perfect, it would be in my hands whenever I need it. But it isn't, and therefore the perfect teapot clearly does not exist. 

And if even the perfect teapot doesn't exist, I see no reason to say that the abstract concept of perfection itself – a Spinozan notion of God – also has to exist in any sense beyond a mental construct. Clearly, I can imagine what I think a perfect teapot should be like, but that has no further existence outside my head. There's no reason to think that perfection itself is any different.

So here too, "perfect" in the everyday sense does not mean the same thing as St. Anselm would have it mean. Nobody uses perfect to mean "something which must exist" : indeed, we often use it to describe things which can't exist precisely because they're perfect ! "Platonic ideal" might be one of Plato's better ideas here, if only in the concept : we can conceive of better examples of chairs and circles and virtues even if we can't bring them into being. That's generally how we use the word, to describe something specific in aspect, not the singularity-like God of the Upanishads. 

As far as the existence of God goes, and very much with my agnostic hat one, I think definitions here are of no help whatever. We can conceive of perfect examples of things we fundamentally do understand, like circles. But perfection itself ? That would require understanding all facets of existence, which as imperfect beings we simply can't do and never will. A general understanding of perfection is beyond our limitations : we can no more say that "god's perfection means he exists" than we can say what a perfect dinosaur would be like. The concept may simply be incomprehensible or it may not even make sense at all.

---

That's my idea of a good definition then. It should be specific, flexible, distinct from alternatives, and describe things which occur at a finite rate (even if only conceptually). If a definition forbids itself from ever existing, or would always be true, then it has no use cases and should be discarded. Those kinds of definitions usually twist readily-comprehensible everyday meanings into something convoluted, unproductive, and useless.

I'll stress "usually" a little bit though as I don't say that the extremes don't matter at all. For example, what do we mean when we say we're know something or that we're certain of it ? Usually, that our own belief is well-formed and our confidence is beyond reasonable or routine doubt. We don't usually mean that we have found Truth Itself, that we can state our claim with literally zero chance of it being wrong, and that all unbelievers are evil and/or stupid. 

Like the case of purest selflessness, this kind of concept definitely does have value, but more in the philosophy classroom than the real world. The extreme cases let us frame our own actual beliefs and compare them to those of others, rather than providing useful, workable definitions in themselves. For example, we can all agree on what true certainty would actually mean, but to use the word more practically, we have to scale things back. That's where the discussions start to get interesting, trying to figure out the limits of our own underlying reasoning as well as that of the others in the debate. 

To a very large extent, I think the question of how we use a definition is very much the same as what we think it means at the most basic level. But then, others may have a different understanding. I don't always agree with the alternatives, but trying to figure them out is usually the fun bit.