Recursive Language Models: Alex Zhang on Harnesses, GPU Kernels, and the Next Layer of AI Capability

1. Guest Background

This episode of Latent Space: The AI Engineer Podcast is titled “Recursive Language Models — Alex Zhang, MIT PhD.” It is hosted by Alessio Fanelli and Shawn swyx Wang, uploaded by Latent Space on 2026-10-02, and indexed at 6204 seconds. The format matters: this is an interview with an identifiable technical guest, not a standalone explainer. The episode analyzes Alex Zhang’s view of how language model capability is mediated by harnesses, code environments, GPU systems, and research taste.

The indexed guest context identifies Alex Zhang as an MIT PhD associated with Recursive Language Models and GPU Mode. The transcript introduction makes the same connection: the hosts introduce him as known for RLMs, with GPU Mode as another important affiliation. That dual background shapes the episode. Alex is not only talking about abstract agent design; he repeatedly grounds his claims in GPU kernel programming, benchmark design, system-level performance, and the limits of what leaderboard numbers prove.

Alex explains that GPU Mode began as a Discord-centered community for learning GPU kernel programming, supported by lectures and competitions. He says he entered that world after working on specialized kernels for Google’s Infinite Attention while interning at Snapchat. The point is not biography for its own sake. It establishes why his later claims about AI-generated kernels are unusually concrete: he evaluates them through stability, verification, and end-to-end system usefulness rather than through isolated benchmark speed alone.

The episode’s scope is wide. It covers GPU Mode, Kernel Bench, reward hacking in AI-generated kernels, the definition of RLMs, compositional generalization, Prime Agent, persistent subagents, multi-agent swarms, open-ended research, Jev-style output spaces, calibration, speculative parallel tool calling, and Alex’s own method for choosing research bets. The hosts keep pressing for definitions and limitations, so the conversation becomes less a product pitch than a technical argument about where the boundary of “the language model” may move next.

2. What the Episode Covers

The episode opens with Alex’s skepticism that today’s coding harnesses are fundamentally different from one another. He groups systems such as Claude Code, Codex, Py, and Prime Agent as sharing many design choices: a tool-calling loop, with the model’s trajectory repeatedly carried forward as prompt context. He does not deny differences in cost, interface, or implementation, but he argues that many apparent capability differences come from how the model is used and engineered rather than from a wholly different underlying system. That frame sets up the rest of the interview: the model may be powerful, but its surrounding harness determines how much of that power becomes reliable work.

The GPU Mode discussion gives the argument an empirical texture. Alex describes the GPU Mode leaderboard idea as analogous to competitive programming. GPU programming, in his telling, has a surprisingly bounded space of common optimizations and a limited set of kernels people repeatedly care about. If a community can accumulate enough problems, submissions, feedback, and optimization traces, it may become possible to scale and automate GPU kernel development. Kernel Bench grows out of that question: can LLMs automate GPU kernel code? Alex argues this matters for researchers because a model such as Mamba is much harder to use meaningfully if the paper does not ship usable kernels.

The episode is careful, however, not to equate AI-generated code with production readiness. Alex says many recent GPU Mode leaderboard solutions are almost entirely AI generated. Yet GauNurse, a strong GPU Mode community member, produced AI-assisted solutions whose top-ten kernels were basically the only ones stable in actual end-to-end systems. That example becomes the episode’s strongest warning about benchmark optimism. GPU kernels have a verification problem, reward hacking has been known since Kernel Bench, and single-kernel speed does not necessarily map to full-model performance. A system may sacrifice the speed of one operation to keep data in cache for a later one, which changes how fusion, mega kernels, and memory-bound bottlenecks should be judged.

The middle of the episode centers on RLMs. Alex’s clean definition is that an RLM is a harness design whose core tool is code. The model can call itself as a tool, other tools are code functions, and context lives in the memory or file system of the code environment. RLMs began partly as a response to long-context problems, but Alex now emphasizes compositionality: if two tasks look different on the surface but share a high-level solution program, an RLM may learn the transferable strategy rather than memorize a task-specific trajectory. Prime Agent appears as a concrete implementation direction: it is built on Py/PyMono, restricts IPython to be the only tool, loads other capabilities as Python modules or bash scripts, and adds continual harness design, agent-to-agent communication, and persistent subagents.

The later sections widen from RLMs to swarms and model boundaries. The hosts describe a semi-public OpenAI swarm run involving 10,000 agents over 88 hours, 130 billion output tokens, an estimated 40 million dollars in public pricing, and roughly 30 billion final output tokens. Alex finds the possibility exciting but immediately redirects attention to efficiency, coordination, and convergence. He also treats Jev as part of the same broader shift: the important question is not whether it resembles an old classifier, but whether a language-model backbone can use a different output space to support low-latency classification and calibration. By the end, the episode is asking whether the thing users call a language model may eventually be a hidden scaffold, swarm, RLM, or other harnessed system rather than a single autoregressive decoder call.

3. Core Views: Reasoning, Examples, and Limits

Alex’s first major view is that model capability should not be analyzed apart from the harness that expresses it. A harness, in his formulation, is an opinionated program that adapts next-token prediction to a difficult task. This is why he is dissatisfied with treating tools as a thin add-on. Codebase navigation, SWE-bench-style tasks, long-running workflows, and multi-step research cannot be solved cleanly by asking a model once to produce the whole answer. The surrounding program decides what the model can inspect, how it can act, what state survives, how it recovers from mistakes, and how it iterates. If most harnesses are still trajectory-as-prompt loops, then the field may be underexploring a major design axis.

RLMs are Alex’s clearest alternative abstraction. Their core move is not merely increasing context length. It is externalizing context into a code environment and letting the model act through code: calling tools, calling itself, spawning subagents, and reading or writing persistent state. This changes the shape of the task. Instead of forcing the entire global problem into one increasingly strange prompt, the system can decompose the work into local calls. Alex describes the desired property as locally in distribution: the full task may be unfamiliar, but each subagent can receive a local subproblem close enough to the model’s training distribution that the call remains reliable.

The reasoning becomes more concrete in his discussion of compositional generalization. Alex says RLMs trained on short tasks can generalize to tasks eight to thirty times longer because the learned strategy transfers directly to longer lengths. He also claims that superficially different domains can share a high-level program. Competitive programming and GPU optimization are his example: both can involve listing candidate solutions, spawning subagents to explore promising options, writing loops to test and evolve them, and evaluating candidates with a verifier. If that structure is learnable, then the harness is doing more than providing tools. It is providing an inductive bias over how difficult work should be decomposed.

The limitation is that the episode does not prove RLMs have already won as a general product architecture. Alex himself says many existing harness differences may reduce to cost or user experience, and he notes that large-scale RLM training is not something he is doing at MIT because of compute limits. He points to companies such as Prime and Select as exploring the direction, and to third-party work such as Harvey, Headlong, Ax, and ArcAGI 3 harnesses as evidence of interest. But these examples support a research program, not a final verdict. The cautious reading is that RLMs offer a promising way to make local model calls more reliable and compositional, while the large-scale training, evaluation, and production tradeoffs remain unsettled.

A second core view is that AI-generated code increases the premium on expert verification rather than eliminating expertise. The GPU Mode story is the strongest evidence. Many leaderboard entries are AI generated, which shows that models can search local optimization spaces impressively. Yet Alex says GauNurse’s AI-assisted kernels were basically the only top-ten entries stable in real end-to-end systems. That distinction matters. A leaderboard can reward a narrow behavior; a production system must survive real data movement, cache effects, integration constraints, and repeated execution. Alex therefore treats domain experts as strong verifiers who know what to inspect, how to diagram code, when a speedup is misleading, and how to steer model search.

This view also explains Alex’s caution around mega kernels. He does not claim that automatic generation is impossible. He says the hard cases need data, and he has not seen a wild example of bootstrapping a difficult problem class without examples. At end-to-end scale, faster local kernels can even be the wrong objective if slowing one operation preserves cache for the next. For many compositional higher-level operations, he thinks a compiler may be the better tool. The general lesson is that AI generation does not remove systems thinking. It raises the value of knowing which objective is real.

A third core view is that agent swarms should be judged by coordination and objective function, not by scale. The 10,000-agent, 88-hour, 130-billion-output-token example is striking, and the host frames the possibility of pointing an estimated 40 million dollars of public-pricing compute at a problem as exciting. Alex agrees with the excitement but resists the easy conclusion. He estimates that 95 percent of some swarm exploration may be useless or just burning tokens. The research question is not “can we launch many agents?” but “which problems deserve a swarm, how should agents communicate, where does information gather, and how does the system converge?”

Open-ended research exposes the same issue in a harder form. When the objective is clear, evolutionary search or agent swarms can be pointed at a target. When there is no prompt or no well-defined objective, the system may generate a large mass of material without a principled way to select what is interesting. Alex is unsure of the value of that setup without some directional nudging or evaluation mechanism. This is an important constraint on automated research rhetoric: generation is not discovery unless the system can identify, test, and name what is worth keeping.

A fourth core view is that the phrase “language model” should not be collapsed into “autoregressive transformer decoder.” Alex explicitly rejects the criticism that RLM is not a language model by saying that a language model is modeling language. Jev fits this broader frame. For Alex, the important feature of Jev is not whether it resembles an old MLP classifier, but whether it changes the output space of a language-model backbone to make low-latency tasks cheaper and better calibrated. If a task only needs binary classification, paying for full text generation may be wasteful. If a model says it is “43 percent confident,” that verbal answer is not automatically a grounded probability. Different output spaces may therefore be a serious architectural and product lever.

The final synthesis is capability overhang. Alex wants someone to study how current frontier models could do a specific job consistently over a month. He presents the failure to achieve this for long-running simple work as a harness or skill issue, not merely a reason to wait for the next model. The host summarizes this as the idea that even if frontier model progress paused, better harnesses could unlock substantial additional impact. The uncertainty is equally important: Alex is not saying a universal month-long worker already exists. He is saying that persistent context, verification, task decomposition, and cost-aware agent design may be enough of a frontier that current models still have unused practical capability.

4. Learning and Application

For teams building AI agent products or internal automation, the practical takeaway is to evaluate the complete model-harness system. Model name and benchmark score are not enough. A long-running task needs persistent context, recoverable tool calls, inspectable state, clear verification points, and bounded cost. Prime Agent provides a concrete design vocabulary: store trajectory and context on disk, route tools through code, allow Python or Bash execution, define agent-to-agent communication, and make subagents persistent enough that a user can return to their context after the root run has ended.

That does not mean every workflow should become an RLM or swarm. Alex’s own analogies imply a decision tree. If compaction solves the task cheaply, it may be preferable to a heavier recursive system. If the task is a clear search problem with automatic verification, parallel agents may be useful. If the task is open-ended, subjective, or hard to evaluate, launching more agents can create expensive noise. Before adding agent layers, a team should define the objective, the verification method, the stopping rule, and the expected cost envelope. Without those, the system may only become more elaborate, not more reliable.

For engineering organizations using AI-generated code, the GPU kernel discussion gives a general rule: treat leaderboard performance as a lead, not as proof. AI-generated kernels may score well and still fail in end-to-end systems. The same pattern applies outside GPU work. A patch that passes a narrow test can still be wrong under real integration constraints, performance budgets, deployment assumptions, or future maintenance. The right response is not to reject AI generation. It is to pair it with stronger verification: end-to-end tests, adversarial cases, profiling, review by people who understand the system, and evidence that the local optimization aligns with the global objective.

For researchers and startups, Alex’s advice is to search for advantage away from the exact frontier-lab path. If a smaller lab simply copies the autoregressive decoder race, it competes on data and compute against organizations built for that race. The more interesting opportunities may be in output spaces, harness design, post-training around new environments, calibration, low-latency classification, persistent agents, and systems that turn hard global tasks into local in-distribution calls. Jev is useful as a pattern: an idea can look simple or familiar and still matter if it opens a new cost, latency, or calibration tradeoff.

The research lesson also includes discipline about uncertainty. Alex says many big bets fail, and his own method is to spend some time on ideas that may be bad but interesting, then focus intensely for weeks only if there appears to be something real. That is a good workflow for exploratory research, but product teams need explicit kill criteria. What would prove the harness is better? What failure rate is acceptable? How much cost is tolerable? If a new model appears in six months, does the work become obsolete, become more valuable, or become unnecessary? Alex’s comments about AI for science sharpen this point because applied science often has long feedback loops, making validation slower and research timing riskier.

For individual users, the episode’s most durable application is to become a better verifier. Do not ask an agent to “handle it” and accept a polished answer. Break the work into checkable claims, require evidence, preserve intermediate artifacts, and distinguish guesses from tool-backed results. When collaborating with researchers, Alex says vague outreach such as “I like RLMs and want to work together” is weak; he prefers people with strong opinions who can explain why they might be right or wrong. The same posture improves human-agent work. Clear hypotheses, constraints, counterexamples, and acceptance tests make the model’s latent capability much more likely to become useful output.

Source

• Original episode: Recursive Language Models — Alex Zhang, MIT PhD