What a Neuron Explanation Must Prove: Curt Tigges on Interpreting Models from the Inside

1. Guest Background

Curt Tigges is a member of technical staff at Goodfire, described in the supplied guest background as a company working on mechanistic interpretability of neural networks. He joins The Cognitive Revolution to discuss Neurons as an Interpretable Basis, the paper he coauthored with Nick Camarata, and the practical research process behind its argument. The episode examines whether researchers can study existing MLP neurons directly and use language-model agents to develop explanations that survive experimental tests.

His relevant experience here is his involvement in that work: investigating neuron activations and connections, developing the Neuron Scientist approach, and evaluating explanations through interventions in the model being studied. This gives the conversation a concrete experimental subject. It also makes attribution essential. Tigges’s accounts of what the team observed, his proposed mechanisms and his ideas for future applications carry different evidential weight. The supplied material supports his role in this research, without establishing a broader career history.

Tigges’s central claim is deliberately narrower than a declaration that interpretability has been solved. Individual MLP neurons, he argues, are more understandable than the field previously believed. Even when a neuron has several meanings, those meanings often form a compact set. He also calls neurons complicated and difficult to interpret. The episode’s substantive question is how to turn that qualified optimism into explanations that predict activations, withstand intervention and eventually help explain behavior.

2. What the Episode Covers

The interview starts by establishing the computational object under investigation. The model’s name varies in transcription, appearing as Qwen or Quinn 3.6 35B-A3B. The program describes roughly 35 billion total parameters and three billion active parameters. Tigges gives a layout of 40 layers, 256 experts per layer and 512 MLP neurons per expert, with eight experts selected at each layer. That produces approximately five million MLP neurons. These are specifications reported in the episode, rather than independently verified model documentation.

Architecture matters because the neuron is already a small computational structure. Tigges describes separate up and gate projections in SwiGLU, capable of recognizing different aspects of an input. The gate path uses SiLU, and the two results are multiplied. His illustrative conjunction combines a cooking context with an ingredient such as garlic. Both paths can affect the resulting strength, so the explanation should not rigidly assign permission to one and magnitude to the other. SwiGLU names the overall arrangement; SiLU is the activation function within it and permits small negative outputs.

Meaning also depends on polarity and readership. Tigges says positive and negative activation can express different things. In another example, a neuron associated with oral health and marine wildlife has different downstream readers in the two contexts. According to his account, the surrounding text activates the appropriate readers in advance. A description based exclusively on the topics of highly activating passages could therefore miss how the signal is used.

Neuron Scientist investigates such questions through an iterative workflow. The team collects thousands of activating examples, then groups them using input connections, textual similarity or patterns of other neurons activating alongside the target. The agent proposes a hypothesis that can contain one meaning or several. It can write and revise sentences, compare near-identical inputs, query actual activations and perform steering interventions. A separate system evaluates its predictions and returns feedback for revision.

A memorable example concerns a neuron that looked incoherent within a roughly 50-token window. Activating passages spanned unrelated subjects. Tigges says investigation revealed a structural relationship: the neuron responded to the second item of a list whose contents had appeared around 1,000 tokens earlier in another format. The explanatory variable was outside the local excerpt.

The primary evaluation asks another model to use only the resulting description to assign each token an activation score from zero to nine. Researchers completely ablate the neuron, patch in those predictions and measure recovery. Tigges confirms that the reported 74% refers to damage-weighted KL-divergence recovery, giving more weight to cases where ablation substantially affected predictions. Supplementary tests include generating text that exceeds the highest activation in the original ten-billion-token dataset. He also reports rates in the nineties on the easier question of whether a neuron activates anywhere in a sequence. These tests probe different capabilities and should retain their separate interpretations.

3. Core Views: Reasoning, Examples, and Limits

The first conceptual shift is from assigning semantic purity to specifying a response function. Tigges questions how much a strict monosemantic-versus-polysemantic distinction tells researchers. A neuron responding to insects’ food, habitats and egg-laying locations could represent one category or three, depending on the observer’s taxonomy. Clothing, exercise equipment, dishes and tablecloths might look unrelated until considered as garage-sale items. That possible unification is an illustration, not permission to invent a convenient story for every collection of examples. The useful obligation is to enumerate the response conditions and test them. Tigges adds that distinct meanings can have distinct upstream neuron subsets, offering a route from descriptive categories to subcircuits.

A second shift concerns what the neuron does after recognizing something. Tigges’s verb metaphor emphasizes downstream effects: increasing or reducing confidence, waiting, checking again, continuing a line of reasoning or changing approaches. He also describes neurons associated with numerical mistakes and off-by-one errors. This perspective asks for an account of the next computational operation. Knowing that a neuron appears in reasoning passages leaves open whether it detects a situation, changes the model’s response to it, or participates in both.

The same distinction helps answer a capacity objection. If neurons have relatively coherent functions, where do the model’s many facts fit? Tigges locates facts in connections between neurons and illustrates how ten interconnected neurons could express substantially more than ten facts. He reports finding many such implementations in MLPs, while explicitly declining to establish that every fact resides there. His observation that examined experts often concentrate on one to three topics—such as jurisprudence, algebra, geometry, timekeeping or medical subfields—provides evidence of local organization. It does not establish a complete map of expertise across the model.

The sparsity results supply a functional argument, but their interpretation depends on the intervention. Historical-percentile ablation keeps a neuron’s extreme positive and negative activations and replaces the rest with its mean, reportedly close to zero. Per-expert top-k instead retains selected signals in the current computation and sets the others to zero. Keeping 64 of 512 amounts to approximately 87.5% activation ablation. The former compares a signal with its own history; the latter selects within a currently active expert. They answer different questions.

Tigges says coherent meaning fades at different percentiles for different neurons and layers. Top-k can preserve a few weaker signals that remain useful, sometimes producing better performance. Percentiles also inherit the reference dataset’s limits: the team extracted examples from ten billion tokens, whereas he estimates the training distribution contained several trillion. A signal’s historical extremity is consequently an imperfect substitute for its causal relevance in a particular computation.

In the hosts’ and guest’s discussion of the tables, removing the middle half of activations usually caused little change. Removing the middle 80% or 90% caused greater losses, although some ARC-AGI tests improved slightly. Tigges suggests faster answering and greater confidence as explanations for some improvements. Those are proposed mechanisms, rather than causes established by the score changes themselves. Initially, sparse reasoning could loop or repeatedly self-check. He corrects the recovery-training quantity to 100 million tokens and says the run took roughly a day on a few GPUs. Before fine-tuning, pass@4 or pass@8 already matched the original model, encouraging his hypothesis that calibration had been disrupted while underlying capabilities remained. Multiple attempts succeeding, however, does not establish reliable single-attempt performance or identify the precise failure mechanism.

Tigges reports gains of five to fifteen points on several benchmarks after recovery and restored results on tests including MedQA and GPQA Diamond. He preserves important uncertainties: the team had not exhaustively tested millions of facts, and his positive personal experience did not establish long-term user equivalence. The evidence supports investigating sparse computation further; it cannot establish universal preservation of capability.

Double-negative SwiGLU activations expose an especially revealing gap. Both projections can be negative while their product is positive. Tigges reports that removing these activations sometimes damages reasoning or memory, yet reversing their token order—or substituting corresponding activations from another sequence—can restore reasoning. The team does not know why. This result cautions against assuming that a convincing textual interpretation captures every computational contribution of an activation.

Finally, interpretable parts do not automatically yield a theory of the whole. Tigges describes attribution graphs and neuron communities, including a few dozen neurons connecting artists with Impressionist, Neo-Impressionist and Post-Impressionist categories. Such examples make intermediate circuits tangible. They leave unresolved the higher-level principles needed to predict behavior in unfamiliar situations. He also limits the case for prioritizing neurons to suitable locations such as MLPs. Attention and the residual stream remain important problems, with SAEs or other tools potentially useful. The episode’s strongest conclusion is a workable experimental entry point, accompanied by clear unfinished business.

4. Learning and Application

A practical way to use this work is to build a small investigation whose explanations can fail. Tigges encourages extracting activations from local models, identifying and steering neurons, and constructing attribution graphs. He also credits earlier neuron-interpretation work, including Transluce. A researcher could begin with a limited set of neurons, record their response conditions and turn each proposed description into predictions on new passages. The immediate deliverable would be a tested local explanation rather than a complete model atlas.

The long-range list example suggests useful experimental designs. Keep much of the vocabulary while changing item order, list formatting or whether the earlier introduction exists; alternatively, preserve the structure while changing the subject. These are applications derived from the interview’s method, not additional results reported by the guest. Their purpose is to distinguish thematic recognition from sensitivity to structural relationships. Minimal changes help discriminate hypotheses, while broader held-out examples are still needed to reveal conditions those experiments miss.

Evaluation should travel with its interpretation. A report using activation patching should state the complete-ablation baseline, how predicted values were produced, what output-distribution difference was measured and whether recovery was damage-weighted. Without those details, the episode’s 74% can easily become an unsupported claim to understand 74% of model behavior. Generating unusually activating text and detecting sequence-level relevance can strengthen confidence in a description, but neither replaces the more specific question of whether predicted activations restore the affected computation.

For sparsification, first establish what is being removed. Tigges describes token-dependent selection of activations: neurons and connections remain available when relevant. An activation-ablation percentage therefore does not establish equivalent permanent parameter removal, storage reduction or inference acceleration. An engineering evaluation should separately compare the original model, the initially sparse model and the recovery-trained version. It should also distinguish single-attempt success from repeated sampling and record answer-time constraints. Otherwise, improvements achieved under one condition can conceal degradation under another.

Activation budgets should then be tested by workload and context length. Tigges reports evaluation up to 32k tokens, especially for coding, and says reasoning and coding require more retained activations than storytelling, potentially by an order of magnitude. That supports measuring different task requirements. It leaves million-token conversations, accumulated interactions and transfers across context windows as open conditions. A system intended for those settings needs evidence from them before relying on thresholds established in shorter tasks.

Safety monitoring requires a similarly explicit scope. A neuron responding to reward-hacking text may be a useful candidate observation point for a subtype of the behavior. Tigges and the host caution that other forms may follow different paths. Researchers applying the idea should examine varied behavior-eliciting contexts and combine observations with ablation, steering or comparisons across training changes. Tigges proposes connecting such bottom-up circuit work with top-down persona steering and anomaly debugging. That combination provides a research strategy for relating internal signals to behavior, without yet guaranteeing comprehensive detection.

Finally, allocate explanation effort according to unresolved questions. Tigges reports a most expensive single-neuron analysis costing about $45 in inference, with typical cases much cheaper, and suggests a coarse pass followed by intensive work on difficult cases. A practical implementation could prioritize descriptions that repeatedly fail prediction, contain several response regimes or concern behavior of particular interest. The tradeoff is that inexpensive screening can conceal complexity, so difficult and unexplained cases should remain visible. One unusually expensive case cannot establish the average cost of interpreting an entire network.

Source

• Original episode: The Neuron Basis: Goodfire's Curt Tigges on Agentic Interp, Monosemanticity & Reverse Engineering