Robot-Use Agents and the Case for General-Purpose Robotics Models

1. Guest Background

This episode of the Y Combinator Podcast / YC Paper Club, titled “Robot-Use Agents: Why General-Purpose Models May Win in Robotics,” was uploaded by Y Combinator on 2026-09-26 and runs 1,789 seconds. The host frames the discussion around a shift that has become hard to ignore in AI systems: coding agents have generalized across software domains more strongly than many expected, and frontier researchers are now showing that related agentic capabilities may apply to robot control. Philip Isola’s robot-use agents essay supplies the episode’s conceptual starting point: if a general-purpose model can learn to use many external tools, perhaps robots are another class of tool that such models can operate.

The identifiable guests are Hamid, Vincent, and Jay. Hamid and Vincent represent Waddle Labs, while Jay is introduced as co-founder of RoboCurve. Waddle Labs works on LLMs that control robots, with three connected pieces: building a harness that lets LLMs control robots effectively, collecting data, and using that data to train better LLMs. RoboCurve is described as an evaluation company for physical AI, measuring many combinations of robots and models: LLMs, classical approaches, VLAs, world action models, and embodiments such as hands, grippers, arms, humanoids, and quadrupeds.

That pairing gives the conversation two useful perspectives. Waddle Labs is focused on the operating layer that lets a model perceive, plan, command, and eventually consolidate robot behavior. RoboCurve is focused on how physical AI capabilities are evaluated across model types and bodies. The host also notes that both groups had recently circulated demos showing LLMs unscrewing caps, coordinating multiple robots, and uncapping a pen. In the episode, those demonstrations are not treated as proof that the field is finished. They serve instead as evidence that the center of gravity may be moving: from narrow robot action models toward systems in which general models write code, call tools, observe failures, and reuse skills.

2. What the Episode Covers

The discussion starts with RT-2. Jay describes it as one of the earliest successful approaches for using AI on robots: a language model pretrained on web text and images is fine-tuned to control robots. The key change is that the model does not output English. It outputs an end effector pose: coordinates and orientation that can be translated into robot joint commands. This matters because RT-2 showed that useful robotics capability could come from outside robotics datasets. Web images and web text improved the VLA compared with training a robotics foundation model without that kind of pretraining.

RT-2 also becomes the contrast case for the episode’s broader argument. Francois compares earlier VLA-style action output to pre-chain-of-thought language models solving simple math: the model had to emit the answer directly, without allocating extra computation to harder cases. In robotics terms, a system that must immediately output an action has limited room to reason differently for easy and complex tasks. Newer systems can look at images, use tools, write code, inspect the environment, and only then act. The control loop begins to look less like a single learned mapping and more like an agentic process: observe, reason, generate a program or tool call, act, and observe again.

The episode then turns to code as policies. The host places that work around late 2022, before today’s coding agents were widely mature. Jay explains the basic design: provide the model with a library of robot-control functions, such as picking up an object, lifting, or moving to a pose; then let a coding agent write Python code that composes those functions into a policy. The surprising result, in Jay’s telling, was one-shot use. The model did not need substantial additional robot data to use those functions, because it had already learned from code data how to sequence operations, use conditionals and loops, and combine tools.

The middle of the episode broadens from robot policies to learning mechanisms. Francois describes in-context learning as appending state-action-result or state-action-reward tuples back into the policy’s active context. It is cheap because it does not require SGD or training FLOPs, and it can improve behavior quickly in low-data regimes. But he also reports important limits: in his experiment, gains were non-monotonic, sometimes improving and worsening aggressively, and they saturated after roughly 20 to 40 examples. Beyond the model’s effectively usable context length, adding more information can make performance worse because the model cannot attend to everything. That leads the conversation to RAG, LoRA, full SFT/RL, and the compression of experience into tools and skills as more durable learning layers.

Waddle’s harness is presented inside that learning hierarchy. A Waddle speaker describes the harness as a form of domain specificity. When a robot enters a new environment, a large model can learn in context and perform the first-pass exploration. But future agents should not pay the same reasoning cost repeatedly. Learned context can be consolidated into programs, skills, memories, or smaller models that run faster inside the harness. The demonstration makes this concrete: Astra uses camera inputs and tool calls to control robot arms that pick up a block and put it into a bowl. Direct control works, but visible latency appears when Astra remains in the loop for every step.

The final technical layer is representation. The host introduces Philip Isola’s Platonic representation hypothesis as the idea that models trained with different data and objectives may converge toward consistent mappings of the world as data scale grows. Jay applies this to robotics: if a very strong language model and a very strong robotics model learn similar world representations, then a sufficiently strong language model may also become a strong robotics model, assuming the hypothesis holds. He also offers a concrete, but still speculative, explanation for Astra’s spatial-intelligence jump: stronger vision and more computer-use and CAD pretraining. Moving a cursor to orbit a CAD object in Blender is not robot control, but it can teach top-down, left-right, viewpoint, space, and object relations that matter for robotics.

3. Core Views: Reasoning, Examples, and Limits

The episode’s strongest view is that robot control may increasingly become a problem of enabling general-purpose models to use robots, rather than building robot-only models that depend mainly on robot data. The reasoning has several parts. RT-2 already showed that web text and image pretraining can improve a VLA. Code-as-policies work showed that knowledge learned from code can transfer into robot function composition. Modern models add better tool use, code writing, and in-context exploration. Taken together, these capabilities make the robot look less like the sole object of training and more like an external system that an agent can learn to operate.

The argument is not that VLAs are useless. It is that direct action output creates a bottleneck. Francois’s math analogy clarifies the issue: if a model must immediately output an answer or action, it cannot easily spend more compute on harder cases. Code policies change the structure. A model can emit a function or strategy that maps state to action, rather than only learning a direct transductive x-to-y mapping. Waddle’s examples make that engineering point concrete: repetitive steps such as approaching or picking up an object can become deterministic code, while variation points such as detection, failure checking, and recovery can call a VLM or LLM inside a loop.

The data argument is equally important. The episode suggests that useful robotics knowledge may come from CAD, Blender, GUI interaction, computer use, and egocentric video, not only from teleoperation traces. Jay’s explanation of Astra is careful: he says its spatial-intelligence improvement may come from stronger vision and more computer-use or CAD data. The claim is not externally verified inside the episode, but it is plausible within the conversation’s logic. Human computer interfaces often encode physical-world metaphors: files, folders, spatial canvases, 3D rotation, camera viewpoints, and object manipulation. A model that learns those digital environments may pick up representations useful for physical action.

The episode also keeps uncertainty visible. The title’s suggestion that general-purpose models may win is a framing claim, not a measured universal fact established by the evidence here. Jay’s statement that some people in frontier labs and robotics foundation-model companies believe general-purpose robots may arrive within two years or earlier is likewise reported as a belief held by some insiders. Its definition is bounded: a robot that can follow natural-language instructions and do what a competent teenager could do with bare hands, including generalization to unseen tasks and environments. That is ambitious, but it is not the same as an unlimited science-fiction robot.

The practical limits are serious. Directly keeping Astra in the loop at every step is slow and economically unattractive. In-context learning is powerful in low-data settings, but it can be unstable, saturate quickly, and break down when context grows beyond what the model can effectively use. Skill accumulation creates its own systems problem: if every deployment adds traces, tools, and policy fragments, the robot stack must decide how to prune, organize, retrieve, and refactor them. The sleep, DAgger, and DreamCoder analogies are useful because they name the missing layer: experience has to be compressed into something reusable. The episode does not claim that this layer is solved.

4. Learning and Application

For robotics builders, the first lesson is to design the system boundary around a harness, not only around a model. A useful robot-use agent needs camera inputs, robot APIs, tool permissions, skill storage, failure detection, memory, retrieval, and policy execution. A general LLM can be valuable for exploration, planning, code generation, and exception handling. But repeated actions with tight latency requirements should be compiled into code, cached skills, or smaller models that execute quickly inside the harness.

The second lesson is to choose the learning mechanism according to the phase of deployment. When a robot first enters a new lab, kitchen, warehouse, or manipulation setup, in-context learning can be a good first tool: it is fast, cheap, and can incorporate a few state-action-result examples directly. But it is not persistent learning by itself. As examples accumulate, teams need retrieval, skill libraries, LoRA, SFT/RL, or offline compression. Interaction logs should therefore be designed as future learning assets, not merely as debugging artifacts.

The third application is to treat code as policies as a software architecture pattern. Teams can expose low-level operations such as pick, move, detect, retry, and verify through stable APIs, then let the model compose those functions into task policies. The stable parts of the policy graph can run deterministically; the uncertain parts can call a VLM or LLM. The tradeoff is that robotics becomes even more like production software engineering: API design, versioning, observability, permissions, and error handling matter as much as model quality. The upside is better control over latency, reuse, and failure boundaries.

The fourth lesson is to widen the data strategy without losing discipline. The episode gives a reason to examine computer-use data, CAD interaction, Blender manipulation, GUI workflows, and egocentric video as sources of spatial and procedural knowledge. But it does not prove that any non-robotics data will automatically improve deployed robot control. The RoboCurve perspective matters here: capability should be measured across models, embodiments, tasks, and failure modes. A model that manipulates a simulated CAD view well still has to prove that the representation transfers to physical sensing, contact, grasping, recovery, and safety constraints.

For strategy, the episode suggests replacing the question “Are general-purpose robots imminent?” with a set of testable questions. Can the system follow open-ended natural-language instructions? Can it adapt to unseen objects and environments? Can it detect failure and recover? Can it reduce latency enough for the task? Can the economics work when reasoning costs are included? Can the skill library grow without becoming unmanageable? If robotics has a ChatGPT-like moment, the episode implies it will probably come from a stack: strong vision models, code policies, tool use, harnesses, retrieval, skill compression, and evaluation infrastructure. Until those pieces are reliable together, impressive demos should be read as directional evidence, not deployment proof.

Source

• Original episode: Robot-Use Agents: Why General-Purpose Models May Win in Robotics