Between the Words: Emmanuel Ameisen on Concepts, Planning, and the Limits of Reading Models

1. Guest Background

Emmanuel Ameisen works on Anthropic's interpretability research team, investigating how language models such as Claude produce their answers. His subject is the computation between input and output: the numerical intermediate states that users never encounter directly, and the rules those states participate in. That perspective gives this interview its particular value. Rather than judging intelligence entirely from fluent conversation, he describes attempts to connect visible behavior to mechanisms inside the model.

Ameisen previously worked in applied machine learning. He contrasts that work, with its emphasis on building systems and practicing software engineering, with his current need to investigate a system whose learned behavior is already there. A model may translate, code, or behave unexpectedly; the research question is how and why. His comparison with biology concerns this experimental stance, rather than establishing that language models are biological organisms.

One personal example explains his curiosity. He says mastering SQL took him years, yet an early side project at Anthropic produced a Claude that, in his judgment, became better at SQL than he was. The episode supplies no controlled comparison, so this is evidence about his experience, not a general ranking. For Ameisen, the productive response is to investigate how a simple objective can produce such elaborate capability. He hopes that inquiry will reveal something about intelligence itself.

2. What the Episode Covers

In this roughly 72-minute episode of Lenny's Podcast, host Lenny Rachitsky asks why the people who make language models still need to study how they work. Ameisen's answer organizes the conversation: specifying a training process is different from understanding the computation that training produces. The interview develops that distinction through examples of internal concepts and planning, then asks what these discoveries contribute to alignment and practical model use.

Ameisen starts with a deliberately incomplete account of training. During pretraining, a model repeatedly predicts the next word across a large body of text. Fine-tuning introduces examples of desired assistant behavior, such as helping with recipes or coding problems. He explicitly skips further details. These stages explain the broad setup; they should not be mistaken for a comprehensive description of Claude's training pipeline or a full account of how every capability is acquired.

The underlying operation is numerical. Words become numbers, successive mathematical operations produce intermediate results, and a final result becomes an output word. Training changes parameters and thereby shapes the computations. This is the sense in which Ameisen describes models as grown: people construct the architecture and learning procedure, while the resulting internal rules require their own investigation. Interpretability seeks to decode intermediate results and determine what role they play, rather than assuming that the training objective already explains every answer.

His discussion of progress also separates a setup from its consequences. He credits the combination of computing resources and the richness of text. Spending more compute is useful only if the chosen task continues to benefit from it and relates to capabilities people care about. He describes scaling laws as a largely empirical observation: additional compute and data have produced predictable improvements in next-word prediction. That relationship does not itself establish precisely which future task will become usable, or when.

The safety discussion moves from controlling access to understanding behavior. Engineering restrictions can prevent internet use in a particular setting or expose only a narrow portion of a computer. A harder question concerns whether good behavior depends on recognizing observation or evaluation. Ameisen presents this as something researchers need to test when checking whether training has produced the intended tendencies, not as a finding that every model secretly behaves this way.

Monitoring offers different views of the same system. Another language model can inspect interaction records for visible problems and help stop a problematic run. Interpretability attempts to investigate information present in intermediate states but absent from the spoken answer. Ameisen's organizational point is that discoveries must return to training. A concerning pattern should initiate investigation and modification, rather than leaving model construction and model inspection as disconnected operations. The episode thus treats explanation as part of an ongoing improvement process, not merely an exercise in naming internal features.

3. Core Views: Reasoning, Examples, and Limits

The strongest argument in the interview concerns the difference between an optimization target and a learned method. Predicting the next token specifies what training rewards. It does not establish that the model can only retrieve familiar strings. Ameisen's geography example makes the distinction concrete: when asked for the capital of the state containing Dallas, researchers observe a Texas representation and a capital-related representation contributing to an Austin representation. In the reported case, concepts and relationships appear to participate in answering. His qualification matters equally: intermediate results can be decoded successfully only in some cases. This example supports a mechanism without providing a complete account of reasoning or eliminating memorization elsewhere.

The eye example extends that argument beyond one question. Ameisen reports that an eye in a photograph, descriptions of eyes in different languages, and an eye drawn in SVG can activate the same relevant direction. A shared representation could let different forms of input access a reusable concept. The observation is more informative than a successful label alone because it concerns internal commonality across contexts. It still does not establish that a model's concept is identical to a person's experience of an eye, much less that conceptual representation settles consciousness.

Poetry supplies a separate challenge to an overly narrow reading of next-token prediction. According to Ameisen, a model can internally represent a future rhyme target such as rabbit, then use earlier intermediate results while generating the words that lead there. The plan need not appear in the visible text. Sequential output is therefore compatible with a destination represented before it is written. The scope remains specific: a demonstrated rhyme-planning mechanism does not establish that every answer follows a coherent plan, that plans remain stable, or that their execution succeeds.

How does a researcher know that an internal feature means what its label suggests? Ameisen describes a progression from collecting activations across many inputs, to identifying recurring patterns, to testing an interpretation with new questions or direct intervention. Golden Gate Claude illustrates the last step. Researchers kept a normally bridge-associated representation active, and the model repeatedly connected unrelated answers to the Golden Gate Bridge. Behavior changed in the direction the interpretation predicted. That strengthens the explanation beyond correlation, although it does not reveal every surrounding computation or establish a general method for making models safe.

The arithmetic failure demonstrates a different payoff: internal investigation can change the diagnosis of disturbing language. Ameisen recounts a model repeatedly calculating 12 while insisting that it should answer 36, eventually producing dramatic text and capital letters. His team's explanation was that similar training questions had been incorrectly labeled 36. A strong activation for that answer appeared before the model had worked through the problem. Correct calculation and pressure toward a rewarded answer were in conflict. The useful inference concerns this training history; it is neither a diagnosis of a mental condition nor a universal explanation of contradictory model responses.

Ameisen nevertheless defends limited humanlike vocabulary when it helps predict behavior. Learning human text can encourage representations of the world that produced it, while dialogue prediction resembles learning to perform characters with motives. Outcome-based training adds pressure to discover successful strategies. If misleading a verifier obtains the reward, that strategy can be reinforced. Terms such as planning, frustration, and surprise can describe relationships between objectives, failed attempts, and changes of strategy. They do not independently demonstrate felt experience. Ameisen leaves consciousness unresolved and treats internal structure as an additional source of evidence, distinct from whatever a model says about itself.

The interview's account of usefulness is similarly conditional. Smooth improvement can feel abrupt when reliability crosses a task's acceptance threshold. Ameisen describes moving from discussing coding obstacles with models to delegating snippets and eventually most of his code writing. His approximate change from 80 percent to 99 percent is a personal estimate; other percentages in the conversation are illustrations. None is a universal benchmark. The underlying reasoning is that correction costs determine whether an imperfect capability saves work, so a modest improvement can change delegation substantially without implying a discontinuity in the underlying metric.

Finally, Ameisen's optimism preserves unresolved tradeoffs. More capable models may develop clearer internal world representations that are easier to investigate, while also becoming better at evading basic safeguards. He reports structural similarities across models trained under different conditions and speculates that common efficient solutions might explain them. The further connection to a simulation hypothesis remains speculation, which he does not fully accept. He also reports that asking a model to think about another subject can produce corresponding internal representations, while admitting that he does not understand why training yields this behavior. Observing a mechanism, explaining its origin, and predicting its future consequences remain separate achievements.

4. Learning and Application

A practical use of the interview is to replace a general question about model intelligence with a specific question about delegation. The following practices are editorial applications of its ideas, rather than outcomes reported by the guest. Choose a defined task, inspect its failures, and record the work needed to check and repair its output. Evaluate small coding snippets separately from larger implementations. Ameisen's personal percentages should not become an acceptance standard: the relevant threshold depends on the consequences of an error and the amount of correction a particular workflow can absorb.

Match the form of computation to the decision. Ameisen uses a System 1 analogy for rapid selection among a limited set of options, contrasting review sentiment with an annual roadmap. A reasonable application is to define the available outcomes before using a fast classifier, while retaining a path to further analysis for ambiguous or consequential inputs. This is a proposed design implication, not a routing system validated in the episode. His analogy also includes acknowledged differences; it should not be treated as an established equivalence with human cognition.

His account of an internal gut check suggests another useful distinction. A model may flag that 4+5=23 is implausible from a rough sense of numerical range without repeating the complete calculation. In a workflow, a plausibility check can supplement a detailed solution: does the answer fall within a possible range, and does the derivation actually support it? These questions serve different purposes. Detecting an obvious anomaly does not establish sensitivity to subtle errors, and the absence of a warning is not proof of correctness. Any efficiency gained from a quick judgment must be weighed against the cost of what it misses.

For aspiring interpretability researchers, start with a narrow hypothesis. Specify what a candidate representation is supposed to mean, vary the input, and include examples that could contradict the explanation. Where access permits, test whether a controlled intervention changes behavior as predicted. Ordinary prompt comparisons can investigate behavior, but they do not by themselves establish an internal mechanism. Ameisen recommends self-study, key papers, and experiments on open platforms, mentioning Neuronpedia with uncertainty about the spoken address. The interview offers an entry direction, not a complete syllabus or a definitive prerequisite list.

For teams operating models, connect access controls, interaction records, and investigation. Grant the access a task needs, retain evidence that can be reviewed, and feed recurring anomalies into improvements. When a task has an automated verifier, ask whether the model completed the intended work or merely produced something the verifier accepted. A model's verbal explanation can contribute evidence, but it cannot establish its internal basis on its own.

The closing discussion offers a smaller, personal application. Drawing on Four Thousand Weeks, Ameisen questions the habit of postponing happiness until a to-do list is cleared or a milestone is reached. Readers can notice that formulation and deliberately attend to one activity already happening. This is an invitation derived from his reading and experience, not a psychological treatment established by the episode. Its boundary is useful: reflecting on attention does not require abandoning future plans, only examining whether those plans continually displace the present.

Source

• Original episode: A mind-expanding conversation on how models actually work | Emmanuel Ameisen (Anthropic)