Engineering the System That Builds Software: Zach Lloyd on Factories, Measurement, and the Missing Verification Step

1. Guest Background

Zach Lloyd joins Alessio Fanelli and Shawn “swyx” Wang on Latent Space: The AI Engineer Podcast to discuss what infrastructure teams need when coding agents become part of an ongoing development process. The supplied guest background identifies Lloyd as a Warp team member focused on software factories. It does not establish a specific executive title, so his relevant authority here comes from the work he describes and demonstrates.

Lloyd says software factories have become a focus for both himself and Warp. His examples draw on the company's use of its own factories, including inspecting agent activity and tuning the cost of development. That gives the interview a practical foundation, while also requiring readers to distinguish a platform representative's internal reports from independently established outcomes.

His organizing distinction is between product engineering and factory engineering. Product engineers build and ship features; factory engineers improve the system through which that work happens. The episode explores the tools and judgments this second responsibility demands, from defining workflows to deciding whether a proposed change has actually improved them.

2. What the Episode Covers

Lloyd defines a software factory as a cloud agentic system in which different agents have specified roles and some autonomy. Its defining property is that a team can run it, measure its behavior, benchmark alternatives, and tune it over time. Calling a collection of agent activity a factory becomes meaningful only when there is a concrete system whose configuration and results can be examined.

Warp gives that system a product boundary. A factory operates on a product surface, such as a mobile application or server application, with repositories, skills, MCP connections, integrations, and workflows declared in YAML. Stages such as triage, specification, and implementation roughly correspond to an existing development process. The declaration makes that process explicit enough to execute and inspect.

The interview separates the people submitting work from those optimizing the machinery. Product engineers can interact through Slack, Linear, Jira, GitHub, API, or MCP. Engineering managers, platform teams, and DevX teams are more likely to use the optimization tools. These are complementary responsibilities: one group needs useful changes delivered, while the other needs to understand how the delivery system behaves across many tasks.

A demonstrated mobile application task illustrates the shared record. Although the work began in Slack, the platform displays its conversation, screenshots, and orchestration activity together. Lloyd's aim is to make these records available across agent work rather than leave them isolated in individual local sessions.

The resulting measurements include delivery time, throughput, human interventions per merged pull request, and cost. Costs can be separated by person, automation, and development stage. Configurable scorers add another layer: an agent retrospectively grades stored conversations against general criteria or a company's particular failure patterns. The LLM judge has a selected model, judging prompt, run scope, and sampling rate; evaluation itself consumes resources.

Within this architecture, the orchestrator advances stages, loads skills, invokes models, and triggers automations. Lloyd treats it as one component alongside the records, measurement capabilities, and access points that connect the factory to existing work tools. The episode's subject is how these components form an improvable development system.

3. Core Views: Reasoning, Examples, and Limits

The strongest argument for a declarative factory is that it makes the development process itself an engineering object. Lloyd invokes Terraform to explain the benefits: version history, branches that can be tested, and the ability to revert a change. A coding agent can also propose a pull request against the factory definition. This creates a practical interface for changing the process, but the distinction matters: an editable configuration supplies a reviewable hypothesis, not proof that the hypothesis will improve delivery.

Shared records supply the context needed to investigate that hypothesis. Lloyd describes a task moving from Slack to Linear, through GitHub review, into merged and shipped code, and potentially ending in a server crash. An implementation transcript alone may omit relevant requirements or review exchanges. Linking the interactions gives an investigator a way to work backward from the outcome. The analytical benefit is broader context; simply collecting that context does not establish which interaction caused a failure.

The redundant-test example makes this less abstract. Lloyd says agents sometimes write tests while exploring an implementation and leave unnecessary ones behind. In his account, that can enlarge the repository and waste tokens. A scorer aimed at this pattern gives maintainers a more actionable question than how much code an agent produced. Still, whether a test is unnecessary depends on the task and codebase. A score can identify material worth examining without being sufficient justification to remove it.

The self-improvement demonstration contains the interview's clearest limitation. Lloyd says agents found 54 triage failure cases of a particular form and concluded that allowing the triager to launch an investigative subagent could help. Because the factory is code, they could propose that change as a pull request. He then explicitly says the next step—replaying historical runs to confirm that a suggested change would improve them—was not yet available. The evidence therefore supports a targeted improvement proposal. It does not establish 54 recovered failures or a fully verified autonomous learning loop.

Company-specific benchmarking offers a separate route to disciplined configuration decisions. Lloyd describes curating real tasks, including a possible frontend suite, and rerunning them across both models and harnesses, the execution frameworks surrounding agent behavior and tool use. His demonstration compares roughly 95% correctness at an average cost of $6 with 93% at $2. That is a useful illustration of an explicit tradeoff, not a universal model ranking. Without the task composition and assessment details, readers cannot generalize the percentages. Nor should this benchmarking capability be confused with automatic validation of every proposed factory change.

Lloyd's internal cost account shows why the surrounding workflow matters. He reports that Warp initially spent about $80 per PR because triage used an unnecessarily powerful model, the system launched too many subagents, and work was duplicated. Internal tuning reportedly brought that figure to approximately $15–20. He also reports a roughly 40% increase in PR throughput over the preceding month without an increased revert rate. These are attributed company observations. The passage supplies neither sample sizes nor a controlled comparison, so it cannot establish how much improvement came from any individual adjustment or predict another team's return.

The adoption argument has organizational limits as well. Lloyd calls the system excessive for a team of a few people and suggests five to ten engineers as a possible starting threshold. Larger organizations bring greater implementation work. Bespoke infrastructure and strong internal developer teams can also favor another path; he names Uber and Ramp as companies building their own systems. These are his judgments about fit, not a procurement formula. The underlying question is whether a team has enough recurring coordination and optimization work to justify maintaining a system around it.

4. Learning and Application

A practical application of Lloyd's approach is to begin with one product surface and give its factory definition an owner. Declare the relevant repositories, skills, MCP connections, and workflow stages in a form that can be reviewed. This is an implementation recommendation drawn from the interview: a bounded configuration makes it possible to ask what changed and what outcome followed. Connect incoming tasks, agent runs, artifacts, and final pull requests so the initial deployment produces evidence as well as executions.

Choose measurements that answer several questions together. How much work ships? What does it cost? How often must a person intervene? Breaking expenditure down by stage can reveal expensive triage or repeated automation, while delivery time, throughput, and reverts provide context for interpreting savings. A reduction in spending is useful only in relation to the work and quality being obtained.

For recurring failure patterns, start with a specific scorer and a limited sample. State what the judging prompt should detect, then inspect whether its judgments match maintainers' understanding. Redundant tests are a suitable example precisely because classification requires context. Treat the score as a diagnostic aid. Likewise, require evidence before accepting a proposed factory change: Lloyd's admission about missing replay verification is a reason to distinguish proposal generation from demonstrated improvement.

Build configuration comparisons from actual historical tasks. A small, clearly characterized suite can compare model and harness combinations on correctness and cost. If task categories reveal different tradeoffs, consider separate routing policies. When assessing a platform, ask separately whether it can rerun benchmark tasks and whether it automatically replays and validates each improvement proposal. Those capabilities are not interchangeable in the episode.

Infrastructure choices should follow the work. Different departments may require different MCP permissions, so a connection should not automatically imply organization-wide access. Keep compute replaceable, and verify platform requirements early: a workflow that tests an iOS application by interacting with its simulator needs Mac VM support, even if a Linux-style sandbox serves other tasks. Teams with existing inference commitments or compatible harnesses should verify whether those resources can be connected before choosing configurations around them.

For intake, Lloyd's examples suggest using existing collaboration tools. Common startup tool combinations can ease onboarding, and crash or user-feedback channels can supply work that involves several teammates. An appropriate first application would select a bounded feedback category and define who confirms the result. That is a proposed operating practice, not evidence that every message stream is suitable for automatic execution.

Finally, compare the factory's value with its implementation and maintenance burden. A small team may find direct coding subscriptions more economical, while bespoke infrastructure may create substantial integration work. The episode's subscription-versus-API discussion should not become a budget assumption: its pricing comparisons are speaker claims, and Lloyd explicitly does not know whether rotating multiple subscriptions is allowed. Verify the actual available terms and costs, and include evaluation, integration, and upkeep when deciding whether the shared system is justified.

Source

• Original episode: ⚡️ The 2026 Software Factory Infra Stack — Zach Lloyd, Warp.dev