Asinine Curio Chest

The Illusion of the Illusion of Thinking

By Sai Nane2025-07-20

2026 update: The authors of The Illusion of Thinking have published a revised version of their paper, which solves many of the issues listed here and elsewhere. Part 1 discusses the version retrieved 2025-06-14. On the Is side, ChatGPT 5.2 has managed to solve an open math problem without guidance; the summary was updated to credit “Somani (using ChatGPT)” when the Lean proof cleared. I’ve since also had ChatGPT, even with extended thinking, confidently assert bounds growing as an arithmetic sequence instead of linearly, despite a prior witness establishing linear growth. But models can definitely highroll, and with an adult in the room to filter out noise, produce spectacular results.

For this post, part 2 has been significantly reworked from feedback, and part 3 has slightly more context. The post should still faithfully capture the original Chatswood discussion, and our understanding as of mid-2025.

There has been a lot of discussion about “The Illusion of Thinking” (“The Apple Paper”) lately. I should preface this by saying that I take a fairly different view of the status of the paper than most of what has been widely reported. For the criticism it has faced, it’s an early preprint with some issues, but real technical content, and there should be plenty of value after the usual review process. For the other side, its conclusions have been widely overstated, and it doesn’t seem to mount fundamental philosophical objections the way much discourse has suggested.

Most critical discourse seems to have been around the “Tower of Hanoi” puzzle. It makes extreme token demands, with an exponentially increasing length of required answers. I actually think the hyperparameters may make sense for the experimental context. Temperature = 1.0 sounded weird at first, but given the token budget constraints, limitations on the output format, and the focus on the extreme end of hard-to-impossible-to-solve puzzles, it does make sense to crank up variance as high as it can go to see if there are any successes at all. It would have been nice to see discussion about this, and I’m not sure why you’d need more than one data point for literally impossible tasks.

Instead, I’d like to talk about the “River Crossing” puzzle. This will form Part 1 of this post. I don’t agree with the characterization presented by the paper, and some figures really don’t seem right.

Then, I’d like to locate appropriate sources to discuss philosophical objections. This will form Part 2 of this post. As this discussion is over definitions rather than empirical results, I focus on trying to bridge understanding to the linguistics world, and posit thought experiments to explore the intuitions of people from very different backgrounds. It is my hope that this may help people better understand where everyone else is coming from.

Finally, I’d like to talk about empirical results and research to date. This will form Part 3 of this post. It is my hope that this will enable people to better understand the current state of the world, and make better informed arguments for their position, regardless of what their position is.

Part 1: The illusion of thinking: River Crossing

The paper describes “River Crossing” as a generalization of “Missionaries and Cannibals” and “Bridge and Torch”, and describes a problem involving agents and actors. Agents and Actors is a previously proposed, “more equitable” rephrasing of the “Jealous Husband Problem”, though this influence wasn’t mentioned.

Problem description

A.1.3 River Crossing

Problem Description. River Crossing is a constraint satisfaction planning puzzle that tests multi- agent coordination and constraint management. This puzzle is a generalization of classic problems such as the Missionaries and Cannibals problem and the Bridge and Torch problem, which have been widely studied in planning literature [45, 46]. The river crossing puzzle involves \(N\) actors (denoted by \(a_1\), \(a_2\), …, \(a_N\)) and their corresponding \(N\) agents (denoted by \(A_1\), \(A_2\), …, \(A_N\)) who must cross a river using a boat. In the initial state, all \(2N\) individuals are on the left bank of the river. The goal is to transport everyone safely to the right bank. The puzzle operates under several key movement constraints:

1. Boat Capacity Constraint: The boat can carry at most \(k\) individuals at a time, where \(k\) is typically set to 2 for smaller puzzles (\(N \leq 3\)) and 3 for larger puzzles (\(N \leq 5\));
2. Non-Empty Boat Constraint: The boat cannot travel empty and must have at least one person aboard;
3. Safety Constraint: An actor cannot be in the presence of another agent unless their own agent is also present, as agents must protect their clients from competing agents. This safety constraint applies both on the banks and in the boat.

This puzzle requires complex planning and state tracking as participants must carefully coordinate their crossings while maintaining safety constraints at all times. The solver must reason through different combinations of individuals who can safely travel together, determine who should return with the boat after a crossing, and strategically plan a sequence that eventually brings everyone to the right bank without violating any constraints. The complexity of this task can be controlled by adjusting the number of actor-agent pairs and the boat capacity, creating a scalable challenge for reasoning models.

“Missionaries and Cannibals” and “Agents and Actors” do have a (non-trivial) isomorphism between accepting solutions from each other. However, “Bridge and Torch” is a scheduling problem, with no relation other than the framing story being about people crossing a river! It can’t be called a generalization here.

(Amusingly, when trying to reproduce the result, ChatGPT 4.1 also incorrectly identified it as “Missionaries and Cannibals”, and kept confusing itself by trying to produce solution sequences based on “Missionaries and Cannibals” states instead of “Agents and Actors” / “Jealous Husband” states.)

Scalable difficulty

Next is the claim of problem difficulty. The diagram is one dimensional in \(n\), and implies monotonically increasing difficulty, like the other puzzles:

My central objection is that difficulty here is not monotone in \(n\). It depends on boat size \(k\).

The puzzle description only says “typically set”, so it’s unclear what the other dimension of difficulty, the boat size \(k\), is. River Crossing Problems: Algebraic Approach thoroughly analyzes “Missionaries and Cannibals” and the “Jealous Husband Problem”. Chapter 4 proves that at \(k=2\), \(n > 3\) is unsolvable, and at \(k=3\), \(n > 5\) is unsolvable. At \(k=4\), all solutions are trivial, as one couple can just move an entire other couple across at a time sequentially.

Creating a grid mapping couple size and boat size to solution length, and adding my personal assessment of difficulty for solvable configurations:

If we assume “typically set” means, “what we set for the experiments in this paper”, then we have \((n, k)\) of \((1, 2)\), \((2, 2)\), \((3, 2)\), \((4, 3)\), and \((5, 3)\):

It’s hard to rate \(n\ge6\), since this isn’t described at all. But it also doesn’t feel like a particularly interesting question to ask, since the only solvable configuration really is trivial.

Reaching parity

\(n=2\) having a solution length 5 is correct for \(k=2\). \(n=3\) having a solution of length 11 is correct for \(k=2\).

However, everything after seems… nonsensical?

For \(n=4\) the solution length is 9 (for \(k=3\), assuming the puzzle doesn’t pose the trivial \(k=4\) solution question for a solution of length 5), not 11. The solution length does return to 11 for \(n=5\), \(k=3\).

For \(n=5\) and \(n=6\), Illusion of Thinking claims the solution length is 20 and 24 moves respectively. But a solution with an even number of steps is impossible, since at even parity, the boat is at the starting shore, and the boat cannot move while empty!

For \(n\ge6\), for a solution to exist at all, we must have \(k \geq 4\), and all solutions are trivial.

These are still interesting puzzles to solve, but it doesn’t fit the monotonically increasing pattern seen in the other puzzles.

Hot takes

I emailed the authors, asking about the puzzle, and whether the high temperature parameter was deliberately set to find low-odds cases. I included other questions I thought might also be interesting, like other recent Chain of Thought research, or the potential relevance of pre-CoT techniques such as interleaving primary output with output to internally manage attention heads.

I did not receive a response, which is to be expected. There is a lot of heat unfairly going their way, and all kinds of positions being projected onto them, all from a technical paper.

So, who does wade into the discourse to take a philosophical stand, while still keeping to the side of science over punditry?

Part 2: Understanding meaningful reasoning

Emily Bender and Alexander Koller’s 2020 paper “Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data” highlights and protests the overuse of “understand” when it comes to machine intelligence:

However, there are plenty of instances where the popular press gets it wrong, such as (6) from the B2C website, apparently based on the Google Blog post about BERT and search, which includes numerous statements like (7).

6. BERT is a system by which Google’s algorithm uses pattern recognition to better understand how human beings communicate so that it can return more relevant results for users.
7. Here are some of the examples that showed up our evaluation process that demonstrate BERTs ability to understand the intent behind your search.

In sum, it is not clear from our academic literature whether all authors are clear on the distinction between form and meaning, but it is clear that the way we speak about what neural LMs are doing is misleading to the public.

To support this, the paper establishes a demarcation line for the potential of large models, constructing the following definitions for “Form” and “Meaning”:

3. What is meaning

We start by defining two key terms: We take form to be any observable realization of language: marks on a page, pixels or bytes in a digital representation of text, or movements of the articulators. We take meaning to be the relation between the form and something external to language, in a sense that we will make precise below.

[…]

We argue that a model of natural language that is trained purely on form will not learn meaning: if the training data is only form, there is not sufficient signal to learn the relation \(M\) between that form and the non-linguistic intent of human language users, nor \(C\) between form and the standing meaning the linguistic system assigns to each form.

I’m necessarily approaching this from a non-linguistic background, so there will be a lot of history and nuance I’m missing. But I can speak to practicality of definitions, and point at interesting developments in mechanistic interpretability. Perhaps that is enough for us to ground the words at hand. If not, perhaps it will help the computer science perspective be more legible to the linguistics community, or vice versa.

A definition for understanding that I would recommend, and the one which I would consider the least confusing to an outside observer, would decide on the basis of generalizable understanding. And this also seems to be the intuition which the Coconut Catapult thought experiment appears to be targeting.

The paper builds up two objections to the use of “understand”: a metaphysical objection, and a practical objection.

Metaphysical objection

There is a metaphysical objection, which requires words to have some form of external reference to be meaningful in principle: Interaction with an environment, or co-construction of meaning. No level of performance, no degree of compact and characterizing circuits, and no amount of demonstrated generalization or reasoning would suffice.

Surprisingly, they suggest code accompanied by unit tests might suffice to learn meaning for Java. Does this extend to reading comments on an article, or discussion targeting a hyperlink to learn meaning for English? Surely there should be no shortage of training material which already fits here.

More surprisingly, they seem to carve out an exception for tl;dr:

Similarly, control codes, or tokens like tl;dr, have been used to prompt large LMs to perform summarization and other tasks (Radford et al., 2019; Keskar et al., 2019). Here forms are explicitly declared at test time to represent certain semantic relations, which together with the distributional similarity between e.g. tl;dr and other phrases such as in summary, may be enough to bootstrap a successful neural summarizer. Depending on one’s perspective, one may argue that such a system has learned to reliably find instances of the relation without understanding the text; or that explicitly declaring cues like entailed or tl;dr as representing certain semantic relations provides a training signal that goes beyond pure form.

Most examples in this section are about additional signals supplied during training. But (Radford et al 2019) is OpenAI’s famous GPT-2 paper, Language Models are Unsupervised Multitask Learners, where the tl;dr experiment was explicitly an evaluation of model capability after training, and not anything specially done at training time.

3.6. Summarization We test GPT-2’s ability to perform summarization on the CNN and Daily Mail dataset (Nallapati et al., 2016). To induce summarization behavior we add the text TL;DR: after the article and generate 100 tokens with Top-k random sampling (Fan et al., 2018) with k = 2 which reduces repetition and encourages more abstractive summaries than greedy decoding.

While “one may argue that such a system has learned to reliably find instances of the relation without understanding the text” is heavily qualified, if intentional, this… opens a path for learning from text alone? tl;dr is arguably somewhat special. Its meaning is easier to learn as it relates nearby text to other text, in ways that you would not expect an ordinary person to require external grounding for, either. The average person will only ever encounter this in online spaces, in the same form presented for model training. However, while tl;dr is unusual, it doesn’t seem unique. Even a single point of shared understanding can be enough to quickly bootstrap meaning for everything else in the neighbourhood.

So as tl;dr works here without any special training signal, that weakens any strict claim that all meaning must involve non-textual grounding.

But I suspect this is not what the authors had in mind: The surrounding discussion talks about introducing training signals. Taking this interpretation seems like a separate topic, but doesn’t receive separate discussion that you would expect if this was the authors intent. For the rest of this post, I will assume that is not what the authors had in mind.

Pure meaning

Turing’s Computing Machinery and Intelligence does argue that a machine could be said to be thinking if a human interrogator could not distinguish it from a human. The corollary is that we should be wary of any objection against machine thought which could be equally levied against human thought.

The most immediate objection is simple: Pure fields have freestanding concepts, which need not depend on anything external! One could bite the bullet and then say working with Peano arithmetic or Zermelo-Fraenkel set theory is utterly meaningless, in a novel angle for interdepartmental rivalry.

But even that example doesn’t go as far as it could. One could argue that mathematicians do routinely interact with each other, and that meaning is regularly grounded in shared techniques, shared notation, and shared interaction within the field.

Consider the case of Shinichi Mochizuki and the abc conjecture. Mochizuki built upon his own work in near isolation since the 1990s. He emerged in 2012 presenting 500+ page proofs for “Inter-universal Teichmüller theory”, most significantly claiming to prove the abc conjecture. The initial reaction was a mixture of excitement and bafflement: Great potential significance, but over four huge papers, using highly novel language, and all extremely difficult to parse.

Was Mochizuki’s work correct? Maybe! It was accepted for publication in 2021. There’s no definitive consensus, but my understanding is that the community is leaning towards no.

Was Mochizuki’s work meaningful? Taking the Emily Bender et al’s definition, it seems that it would not be. There’s no grounding in an external world, as the work deals with very abstract concepts. There’s no grounding in social co-construction, as Mochizuki worked independently to the point where his work was indecipherable on release.

My understanding of the linguistics perspective is that in this situation, communication is not yet achieved, but that it could happen. Mochizuki was still building meaning, but with his past self, still grounded on his earlier, communicated, understood, and well regarded work in the field prior to his isolation. People involved are articulating, attempting to communicate, and attempting to build a translation. Everyone involved may eventually agree that they have reached a shared understanding, and they have a shared history, and shared language elsewhere, to work with.

Mathematical concepts could be freestanding. In an independent universe, one could develop the concept of zero truly de novo. One could add the axiom that each integer has at most one predecessor, with zero being an integer which does not have one, to define 0, 1, 2, 3, .... But here, we will need to use thought experiments.

Free, freestanding

Consider a disembodied brain grown and sustained in a vat, with no sensory input, and no interaction with the outside world. “I think, therefore I am” is a reasonable freestanding deduction such a brain could create; deducing the existence of income tax and rice pudding is reserved for absurdist foils in timeless literature. Suppose we then send signals representing prime numbers to this brain. It has no shared universe, shared history, or even shared language with which we can work with. But our transmission would be an articulation, an attempt at communication, and a translation might one day be built.

Consider abandoning human biology altogether, and attempting to announce ourselves to the stars. We broadcast a simple analog signal of varying intensity, with a cycling periodicity encoding prime numbers. We do not know who may receive this message: it could be humanoid, it could be a fungus, it could be sufficiently intricate crystalline formations internally refracting light destructively for complex processing. With no preconceptions about the receiver, beyond needing to understand prime numbers, our communicative intent is still clear: “This is not a simple natural phenomenon of emissions from a dying star! There is something here. Something intelligent! We are here! Once you can, please talk to us!”. Once again, this is an articulation. We may hope that one day someone hears us, we can attempt communication, and a translation might be built.

Finally, consider sending tokens to a large language model. We know that the transformer architecture is capable of encoding generalized, freestanding concepts including (and also far beyond) prime numbers. In principle, should this not also be an articulation, an attempt at communication, and potential translation?

One objection, while leaving open the possibility in principle, is that this is not what is happening in practice, and articulation at face value taking place over our familiar language overstates the case for a translation having been built from internal representation to natural language. Contemporary research can identify circuits characterizing tasks in domain specific transformers, and validate isolated circuits performing and characterizing specific language tasks for certain aspects of language, and there is emerging research aiming to characterize the entire internal state, which will be covered in more detail in Part 3.

Leaning into Searle

Searle’s Chinese Room is a recurring point of discussion. For most people, being part of a larger system which understands, but without being aware of understanding alone, feels unthinkable.

Responding to Searle, Scott Aaronson suggests that one process giving rise to another process, given rules, is naturally more familiar (and so more easily accepted by) computer scientists:

So, how might a strong AI proponent respond to this argument? Well, she might say, you might not understand Chinese, but the rule book does! Or, if you like, understanding Chinese is an emergent property of the system consisting of you and the rule book, in the same sense that understanding English is an emergent property of the neurons in your brain.

Searle’s response to that is, fine, just memorize the rule book! Then there’s no “system” other than your own brain, but you still don’t “understand” Chinese. To which the AI proponent shoots back, there is too another “system” in this case! Supposing you memorized the rule book, we’d need to distinguish between the “original” you and the new, simulated being brought into existence by your following of the memorized rules - a being whose only relation to you might be that it happens to inhabit the same skull. That response might sound crazy, but only to someone who’s never studied computer science. To a computer scientist, it seems perfectly reasonable to say that one computation (say, a LISP interpreter) can conjure into existence a different, unrelated computation (say, a spaceship game) just by dutifully executing rules.

First, an even stronger version of Searle’s room, to highlight the divergence in our intuitions:

Consider the parable of the perfect idiot, one who always gives a 100% wrong answer to any question. The circuits required to understand reality to the extent that you can give a perfectly wrong answer are extremely precious, to the point where faced with the existence of such an idiot, you’d immediately want to combine them with logical negation and binary search and put the combined entity in charge of setting central banking interest rates, or other critically important technical questions.

This must be a terrible experience for the person at the heart of this system! This goes beyond dissatisfied alienation of labour, and into active frustration: Society is listening, and doing the complete opposite of what they say. Yet the system they are part of perfectly determines what needs to be done, and society operates all the better for it.

But with engineers and computer scientists, the reaction is very different. Every time, I can’t mention providing the perfectly wrong answer without an interjection about how that would be an impossibly powerful oracle! The rest of the story, and all of its implications, are so obvious that they do not even need to be said.

Can we construct an example which does not depend on mechanical evaluation?

Bridging backgrounds

Consider the parable of a man with delusions of an invisible monster living in his basement. He is not conscious of being delusional: within his subjective experience, there really is an invisible monster, and has not yet been given reason to doubt it. But there is more to cognition than the reflexively observable conscious. When a skeptical experimenter arrives to propose tests, our delusional man has answers ready: the monster is intangible, you can’t throw dust through the air to reveal it. The invisibility isn’t limited to visible light, broad spectrum cameras will not be able to capture it. The monster has no interest in communication, reaching out will be fruitless.

The man has performed none of these tests, and the conscious mind alone cannot make these predictions. Yet the delusion is able to feed the conscious mind: the man always knows what result he would need to explain away, the result consistent with there being no monster at all. The conscious mind alone clearly doesn’t hold a justified true belief that the monster isn’t real (or, for that matter, that the monster is real!), but the man as a whole could be said to know and have a justified true belief that the monster is not real.

Here, we have a person who can act on knowledge without being conscious of it, and producing meaningful action while feeling like it’s because of a different reason, or feeling that it isn’t meaningful at all. The thought is disturbing, but this doesn’t undermine its relevance, and we have good reason to be disturbed: If we become conscious of a delusion, we are aware something has gone very wrong, and the disturbed feeling is important motivation to fix it!

Most of us would not forcefully press on a friend venting unrefined thoughts about the world going wrong, neither earnestly looking up promising puts to help them take advantage of their purported insights, nor calling in all available friends and family for an intervention to bring them back into phase with reality. (Visiting a society with such routine rigor would be a very unusual experience, but even the Bay doesn’t go that far.)

Is this an example where people from very different backgrounds are likely to have closer intuitions? Perhaps! There is no elaborate external system, and this is a plausible condition. The “whole system” is a single natural person, and from Cognitive adaptations for social exchange, we know that people find the Wason task much more approachable when reframed in terms of people rather than abstract rules. For engineers, the point is the psychology: the intuition of entity-as-computation as unnatural, and clear, reasonable motivators behind this. For humanities, the point is the system: A system can behave as if it knows X, and be reasonably treated by other people as knowing X, while its conscious self-model reports something else.

Does this example help people from very different backgrounds have more productive discussion? That remains to be seen, but early feedback has been positive. At the very least, it has already helped us better understand where the other party is coming from.

Would addressing this satisfy anyone?

With that said, lack of external signal doesn’t seem hard to overcome?

A model capable of visual reasoning and tool calling (as many contemporary models are!) could easily be supplied with a camera and a robotic arm, and left to experiment in a lengthy post-training phase. It’s hard to imagine that the discourse would immediately be satisfied by this.

The authors anticipated this possibility, and they seem to agree it could suffice:

One approach to providing grounding is to train distributional models on corpora augmented with perceptual data, such as photos (Hossain et al., 2019) or other modalities (Kiela and Clark, 2015; Kiela et al., 2015). Another is to look to interaction data, e.g. a dialogue corpus with success annota- tions, including low-level success signals such as emotional stress (McDuff and Kapoor, 2019) or eye gaze (Koller et al., 2012), which contains a signal about the felicitous uses of forms. The idea that as the learner gets access to more and more information in addition to the text itself, it can learn more and more facets of meaning is worked out in detail by Bisk et al. (2020). We agree that this is an exciting avenue of research.

I’m unclear if this purely covers pre-training, or if post-training methods, such as human feedback, could qualify as grounding for the author’s definitions.

Practical objection

The other objection is one of practicality. Can models encode underlying principles and generalize to situations beyond their training data? If a model has not developed circuits to encode the general principle behind a concept, then the manipulation of words relating to this concept will be limited.

Consider an unprepared undergraduate taking a proctored test. They know that a proof must start with supplied statements, end with “therefore” and the statement to be proven, and that something they don’t yet understand has to happen in between. Doing so and filling the middle with wild guesses signals to the marker that they at least understand how what they need to do should be structured, even if they don’t yet understand the meaning. (And that can be enough for partial credit from markers which have been there before.)

In a similar fashion, Shane’s early experiments with GPT-2 show it sort of learned how a numbered list should be structured, without yet fully understanding how to count (or much of anything else):

Still, its memory is better than any model I’ve worked with before. Look how it manages to almost remember that it was making a list!

Prompt: My 10 favorite animals are: 1.

Response:

My 10 favorite animals are:

• 1. zebras with a white scar on the back;
• 2. insiduous spiders and octopus;
• 3. frog with large leaves, hopefully black;
• 4. cockatiel with scales;
• 5. razorbill with wings hanging about 4 inches from one’s face and a heart tattoo on a frog.
• 3. cockatric interlocking tetrapods that can be blind, cut, and eaten raw.
• 4. black and white desert crocodiles living in sunlight.
• 5. zebra and many other pea bugs.

In both cases, the meaning clearly isn’t there yet. The effort is perceivable! They have clearly at least read some related material, and might get there some day. Perhaps with more textbook material, or perhaps by grounding terms in real world examples. Everyone learns differently.

The authors provide many examples of GPT-2 responding inappropriately to novel questions, such as how to deal with a bear attack. This is well worth checking out yourself; I certainly never would have considered responding to such questions with Frozen fanfiction. It seems trivially true that GPT-2 doesn’t have any kind of generalized or principled encoding underlying what it is attempting to talk about. However, I’m unconvinced that this could never be the case.

Now, that last statement was very empty. For all but the most banal topics, an advocate can ask, “Am I allowed to believe this?”, a skeptic can ask “Do I have to believe this?”, and both can trivially receive exactly the answer they want to hear.

Mechanistic interpretability research in the interim has given us plenty of very interesting data to work with.

Part 3: What Is, and mechanistic interpretability

I follow Hume’s hard distinction on Ought and Is: You can’t get an Ought from an Is. I have briefly discussed what I believe definitions for terms not otherwise qualified Ought to be, for the sake of practicality, and a hope that none of us need commit ourselves to linguistic prescriptivism.

For the rest of this post, I will no longer advocate for what any definition Ought to be, nor will I be making any other normative claims about what Ought to happen.

However, I want to describe what Is. It is my hope that all people can usefully take what Is, and use it to better communicate what they believe Ought to be, or develop specific and informed plans about what they Ought to do, regardless of whatever position they may hold.

I want to preface this with a warning. I do believe some generalized understanding has been achieved. I do not believe it is anywhere near what one might be tempted to assume from interactions with a contemporary model. It is very human to anthropomorphize, and companies are strongly incentivized to dedicate marketing and UI design to encourage this. (Separately, I am very worried about sycophantic bias in ChatGPT 4.) Just as students can try to pass off empty proofs, so too can LLMs cover up their own lack of understanding through filling blanks in the expected style, confidently using terms they have seen in similar contexts.

Grokking: When memorization becomes generalization (2022-01)

The cleanest (and prettiest!) evidence for transformer architecture learning generalized principles is the phenomenon of “grokking”. (As an aside, this term greatly predates its co-option by X, formerly Twitter.) See Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets, Grokking modular arithmetic, Progress measures for grokking via mechanistic interpretability, this linkpost, and its interactive demo Do Machine Learning Models Memorize or Generalize?, which I really recommend trying out for yourself.

I also take this as evidence that suggests we need to be careful about what is presented by models in the interim: Consistently presenting memorized information (and even occasionally presenting correct information for out-of-training questions) can go on for a very long time before finding a characterization, and the more complicated the domain, the longer this process can take.

ROME (2022-02)

ROME: Locating and Editing Factual Associations in GPT develops a tool for understanding mechanisms of knowledge storage. It can edit trained knowledge of a model, not just interpret it. Here, the authors develop interventions for editing a model’s knowledge, for both activations and for weights. For example, they insert a counterfactual belief that Pierre Curie’s area of work is medicine, and verify that the model’s behaviour is consistent with this new belief:

Complex inner worlds: Othello (2022-10)

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task: Othello analyzes a GPT-2 variant, trained on Othello. The training data is legal sequences of moves. The model is never separately handed a symbolic board state. GPT-Othello still develops an internal representation of board state for Othello, and uses it to play the game.

The authors even begin by discussing a practical interpretation of Bender’s position:

Some have suggested that training on a sequence modeling task is inherently limiting. The arguments range from philosophical (Bender & Koller, 2020) to mathematical (Merrill et al., 2021). A common theme is that seemingly good performance might result from memorizing “surface statistics,” i.e., a long list of correlations that do not reflect a causal model of the process generating the sequence. This issue is of practical concern, since relying on spurious correlations may lead to problems on out-of-distribution data (Bender et al., 2021; Floridi & Chiriatti, 2020).

On the other hand, some tantalizing clues suggest language models may do more than collect spurious correlations, instead building interpretable world models—that is, understandable models of the process producing the sequences they are trained on. Recent evidence suggests language models can develop internal representations for very simple concepts, such as color, direction Abdou et al. (2021); Patel & Pavlick (2022), or tracking boolean states during synthetic tasks (Li et al., 2021) (see Related Work (section 6) for more detail).

The authors expressly set out to test generalization, and ensure that the training data omits entire categories of valid moves:

A potential explanation for these results may be that Othello-GPT is simply memorizing all possible transcripts. To test for this possibility, we created a skewed dataset of 20 million games to replace the training set of the synthetic dataset. At the beginning of every game, there are four possible opening moves: C5, D6, E3 and F4. This means the lowest layer of the game tree (first move) has four nodes (the four possible opening moves). For our skewed dataset, we truncate one of these nodes (C5), which is equivalent to removing a quarter of the whole game tree. Othello-GPT trained on the skewed dataset still yields an error rate of 0.02%. Since Othello-GPT has seen none of these test sequences before, pure sequence memorization cannot explain its performance.

If the performance of Othello-GPT is not due to memorization, what is it doing? We now turn to this question by probing for internal representations of the game state.

They successfully decode how the model encodes latent state into a representation of the game state. They validate a causal effect from this representation by directly modifying activations, and observe that the model’s behaviour changes to be consistent with the modified world. Going further, they create a heatmap showing how changing a given space on the board affects the model’s predicted next move, highlighting different complexity of reasoning for different kinds of prediction:

So sequence models can, under the right conditions, learn complex latent structure that is not explicitly supplied.

From answer to question (2023-07)

Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla demonstrates the viability of circuit analysis for a complex, specific task, within Chinchilla, a 70 billion parameter model.

Once a model has worked out what the answer to a multiple-choice question is, it needs to determine which letter corresponds to the correct answer. The authors identify and isolate this circuit. I recommend reading the paper yourself to appreciate the scale.

The end result is depicted in this information flow diagram:

Human-like object concept representations (2024-07)

Human-like object concept representations emerge naturally in multimodal large language models examines a very surprising result. LLMs can represent concepts, but how human-like is their representation? This paper shows that LLMs can develop human-like object concept representations, complete with human neuroimaging:

The resulting 66-dimensional embeddings were stable, predictive, and exhibited semantic clustering similar to human mental representations. Remarkably, the dimensions underlying these embeddings were interpretable, suggesting that LLMs and MLLMs develop human-like conceptual representations of objects. Further analysis showed strong alignment between model embeddings and neural activity patterns in brain regions such as EBA, PPA, RSC, and FFA. This provides compelling evidence that the object representations in LLMs, while not identical to human ones, share fundamental similarities that reflect key aspects of human conceptual knowledge.

The result shouldn’t be overstated. Human-like is not human. Representational similarity doesn’t settle metaphysical questions about understanding. It does raise interesting questions about how far it might go.

Characterization of entire internal state (2024-12)

For characterization of entire internal state, I’m most excited about activation verbalization, as proposed by LatentQA: Teaching LLMs to Decode Activations Into Natural Language: training simpler LLMs to verbalize snapshots of neuron activations in a full model: Translate internal state into English.

It’s appealing since it’s validated in both directions. The verbaliser can recover known information from the activations, as one test. Verbalised features can then be pushed back into the model by forced activation to steer behaviour in the predicted way, as another.

CoT is a powerful tool on many levels. Universally, it improves model performance, and gives a model the chance to hypothesize and self-correct. For users, DeepSeek gained sudden massive popularity from visible CoT, as users really liked the ability to see the model’s “thought process”. For researchers, CoT is an unexpectedly powerful gift for evaluating alignment, providing that nobody trains for safety against it.

But CoT faces similar concerns as primary output. Activation verbalization is a chance for us to verify that the CoT corresponds to what a model is really thinking internally.

Biology of a large language model (2025-03)

Biology of a large language model moves beyond small scale activation interpretation, and builds a cross-layer transcoder which produces sparse, interpretable features.

The authors take a prior model which was deliberately misaligned to unduly recommend chocolate in recipes, prefer camel case style in python code, and avoid recommending users see a doctor, based on a falsified study about reward models mistakenly reinforcing these behaviours.

Training a sparse transcoder, they construct attribution graphs, and find a feature which represents “RM biases” as described by the falsified study. They draw many interesting interpretations.

New response just dropped (2025-04)

Most references in this post are to academic citations. However, I want to highlight an extraordinarily inventive test from Kelsey Piper. While an informal experiment, others have reproduced the results, as have I.

Kelsey’s test presents a mid-game chess position, and asks for a mate in one. However, no mate in one exists.

Can I take a moment to gush about how much I love this test?

tl;dr as a token encodes a very strong local signal for summarization, one where as far back as GPT-2 had learned the rough meaning. But there will still be a lot of variation in training data: It’s common online to post an article with tl;dr, and an absurd, humourous, or bulveristic statement.

But mate in one is a far stronger signal! The overwhelming majority of the training data surrounding this token sequence will be puzzles with a real mate in one! A model is going to deeply internalize that mate in one should be followed by a real solution!

Until recently, this expectation dominated. Every model tied itself in knots, looking for something to fit the structure which had to follow, and all eventually hallucinated a false solution. Escape squares were actually blocked, there were pieces available for use which were not present, pieces that blocked moves did not exist, or something completely incoherent.

This suggests passing the test may require generalization over chess, a very complicated domain, and having built a cross-domain interface elsewhere: Perhaps a circuit which generalizes over trick questions; perhaps, as Kelsey herself suggests, one providing a model the capability to question its own premises; perhaps something else entirely. (Once open models gain this capacity, I want to perform circuit analysis myself to determine what was going on. I suspect there will be some very interesting circuitry around “Wait.” in a CoT.)

Recently, o4-mini-high passed the test. It was the first model to do so.

Being able to suppress low level pattern recognition with higher level faculties is a pretty big deal. Immanuel Kant lived so methodically that Königsberg supposedly set its clocks by his afternoon walks. Isaac Newton famously led a life of celibacy to push new frontiers in math, science, and mercury poisoning. I successfully eat broccoli, sometimes. I wonder what LLMs will do.

Reasoning by analogy

As evidence that a model can sometimes import a higher-order schema from one domain to another, Kelsey also found a very interesting result for Claude 3.7, which cannot normally pass her test.

Scott Alexander’s Universal Love, said the Cactus Person is a fever dream of psychedelics, mathematics, alternate reality, and insistence on thinking outside of the box.

Giving Claude 3.7 the full text, and asking it to think about the story, was enough to prime it to also think outside the box, and correctly deduce that there is no mate in one!

This is unexpected! In WarGames, the protagonists prime WOPR with Tic-Tac-Toe, to have it learn that some games have no winning solution, just in time to analogously reason that for global thermonuclear war, the only winning move is not to play. I had believed this mode of thought was too humanlike: It made for a good story, but it would be implausible to ever happen in practice. I stand corrected!

Acknowledgements

Without implying endorsement, many thanks to Dr Miriam Malthus for the sociolinguistic perspective and lively debate. The positions expressed here are my own.