The Inference System Behind the Token: Cost, Quality and the Limits of Optimization

1. Guest Background

Dan Fu is introduced as Together's head of research. The supplied guest background places his work in AI inference optimization, specifically first-token latency, time per output token, throughput and goodput under service requirements. That remit explains the perspective he brings: the relevant object is a model delivering a usable response through a production system, with resource consumption and user experience considered together.

His examples establish relevant experience without requiring a broader biography. Dan describes teams inspecting forward passes and optimizing GPU kernels, then explains how Together is experimenting with agents that iterate on those optimizations using production traffic. He also discusses internal model use, accumulated failure cases and agent evaluations. These are the operational experiences behind his insistence that a smaller representation of model weights must still support working tools and completed tasks.

This is a multi-speaker YC Paper Club session rather than an interview confined to Dan. Jeff, speaking about Ollama, examines quality differences between inference endpoints. Joe identifies himself as Luminal's cofounder and CEO and presents a compiler approach. Liangsheng discusses SGLang's caching and memory systems. Their contributions broaden Dan's request-lifecycle account into a discussion of how inference is supplied, selected, compiled and coordinated. The episode is most useful when those perspectives are read together.

2. What the Episode Covers

The title's claim that inference matters twelve times more than training begins as a conditional calculation. Francois cites older DeepSeek figures: approximately 2.8 million H800 hours for training and 43,000 H800 hours of inference per day. Maintaining that daily usage produces his estimate of roughly six times the training compute each year. Assuming a two-year useful life for the weight file produces twelve times. The point is to include repeated serving work in a model's lifetime resource budget; the calculation does not establish a universal cost ratio.

Dan makes that repeated work concrete by following a request through the system. An API request reaches the infrastructure, is routed to a GPU replica, and undergoes tokenization and tool-format handling. It then waits for scheduling and batching, passes through prefill and decode, and returns token IDs that become streamed text. Tool use adds an important correctness boundary: the mathematical model produces a request to use a tool, while an external mechanism must interpret and execute it correctly.

The two principal GPU phases impose different demands. Prefill processes the prompt and generally offers relatively high arithmetic intensity. Decode generates one or a few tokens at a time and is commonly constrained by moving weights from GPU memory into computation. Scheduling consequently chooses between objectives such as accommodating more users and delivering a more responsive experience. Optimizing one headline throughput figure cannot settle that choice.

Francois organizes the available interventions into model changes, kernels and runtime, and the application harness. Dan develops kernels, parallelism and quantization in greater detail. His internal Together Compiler project uses agents to optimize a target model on target hardware against real traffic, although he explicitly says it is not actually a compiler. Quantization supplies a separate deployment lever: his storage example takes a three-trillion-parameter model from about 6 TB in BF16 to 1.5 TB in FP4. That arithmetic explains the incentive to compress weights, while leaving quality as a separate question.

The other presentations extend the system boundary. Jeff describes aggregating providers to combine their strengths across cost, intelligence, speed, region and privacy. Joe proposes controlling compilation complexity by mapping roughly 2,000 PyTorch operations into approximately fourteen or fifteen primitives before finding hardware implementations. He illustrates this by decomposing matrix multiplication into broadcast multiplication and sum reduction. Liangsheng starts from agents sharing long contexts and adding short suffixes. His account of RadixAttention connects prefix reuse, eviction of unused tree leaves and cache-aware request ordering. Together, these presentations explain why inference performance depends on coordinated decisions across the stack, rather than a single faster mathematical operation.

3. Core Views: Reasoning, Examples, and Limits

The episode's central analytical move is to make performance claims conditional on workload and accounting. Francois cites DeepSeek figures of approximately 96% input cache hits and 72-hour retention, alongside prefill throughput of 60,000 tokens per second per GPU excluding hits and 1.5 million including them. The larger figure credits work avoided through reuse. It therefore describes a particular service under a particular traffic pattern, rather than the raw rate at which a GPU newly processes arbitrary prompts. Dan's coding example explains the mechanism: a follow-up request can benefit from routing back to a replica that already holds the codebase's KV cache. Simple load balancing may discard that advantage.

Even cache-hit improvements need a denominator. Liangsheng's move from 95% to 99% hits reduces the uncached share from 5% to 1%. That is a fivefold reduction in the remaining uncached portion, not evidence that total serving expense falls fivefold. Decode, queuing and data movement still belong to the request. Goodput supplies a more useful service objective by asking how much output meets the required service level. First-token latency and time per output token remain necessary companions. Roofline reasoning helps explain the hardware constraint by comparing peak compute with bandwidth multiplied by arithmetic intensity; it does not remove the need to measure the service.

Hardware expansion carries a similar qualification. Dan explains how a larger high-bandwidth domain allows experts to occupy separate GPUs and collect larger token batches. Those batches make the work resemble the large matrix multiplications GPUs handle well. But a replica depending on 72 GPUs also creates a larger failure exposure: one lost GPU can remove the whole replica unless recovery mechanisms keep it functioning. Reliability engineering is therefore part of realizing the throughput opportunity. The episode does not supply a single scaling factor that makes this tradeoff disappear.

Quantization introduces a quality constraint that storage arithmetic cannot answer. Dan's evaluation stack combines public benchmarks, activation KL divergence, hundreds of historical failure regressions, internal use and agent loops. The Flappy Bird coding loop is memorable because it asks whether an observable program still works, rather than only whether a score remains close. Asking a model to optimize kernels tests another form of practical capability. Neither task represents every application, which is why their value lies in complementing other tests. Dan reports that Together's NVFP4 recipe matched the original MiniMax FP8 model in the displayed evaluation. Without detailed scores or a measured throughput increment, that remains a bounded comparison from the presentation.

Jeff makes the quality issue broader than quantization. Identically named models may differ through weight versions, sampling, tool parsing, faulty hardware, custom kernels or KV-cache precision; some problems appear only with long contexts. His fifteen-step example shows why agents warrant whole-task evaluation. If each step independently has a 10% probability of deviation, and any deviation counts as failure, the probability of at least one deviation is approximately 79%. This is illustrative probability arithmetic, not a measured failure rate for coding agents. Its value is to expose assumptions that a single-call comparison can hide, including how a workflow treats and recovers from deviations.

Joe's compiler presentation offers an automation argument with an explicit correctness boundary. An e-graph retains equivalent representations, allowing the system to extract candidates and time them on hardware. Roofline-based pruning can reject candidates whose best possible performance would already be worse than a measured alternative. But equivalence must respect numerical semantics: tree reductions and sequential reductions can produce different floating-point results. High-level metadata and egglog rewrite proof chains provide a way to locate invalid transformations. Budgeted search also does not guarantee a global optimum. Spending hours or a day compiling can be sensible when runtime savings recur, but the percentages Joe offers describe a tradeoff rather than a universal benchmark result.

Liangsheng's overlap scheduling and HiCache reinforce the system-level argument. Preparing another batch while the GPU works can reduce host-side gaps; keeping prefixes across GPU memory, host memory and external storage can extend reuse. He reports hardware-dependent throughput gains and a 56% reduction in a latency metric. Full configurations are absent, and the latter metric is ambiguously transcribed. These details support the mechanisms and the speaker's reported results, while leaving their transfer to another deployment an empirical question.

4. Learning and Application

A deployment team can begin by building its own lifetime compute scenarios. Record training compute separately from daily inference demand and the expected useful life of the weights. Compare outcomes when traffic grows or the model is replaced earlier. Francois's calculation supplies a structure for that exercise, rather than a twelve-times constant to insert into every budget. Define service targets alongside it: first-token delay, generation pace and the volume that actually meets those requirements.

For a coding service, test cache-aware routing with sequences of requests against the same repository. Compare repeated prefill work and total response time against ordinary load balancing, then vary the interval between turns. This makes the retention decision depend on observed reuse. A replica holding the right prefix may also have a longer queue, so the experiment needs both benefits and waiting costs. After improving hits, measure complete-request resource use; the reduction in uncached work cannot by itself establish the financial saving.

Treat quantization as a staged release decision. Check numerical changes and task scores, then run historical failures, tool interactions and complete agent tasks representative of the application. Dan's training-free approach uses calibration data and Scale Search. His reason for avoiding further training is the forgetting his team observed without the complete original training dataset. That is a conditional engineering experience, not evidence that all post-quantization training fails. The practical boundary is that agreement on a few public scores does not establish equivalent behavior for an application's workflows.

For endpoint monitoring, separate historical stability, agreement with reference weights and task correctness. Jeff's next-token log-probability approach can make a lightweight drift probe when the API supports it. A stable but incorrect starting baseline remains incorrect. Reference-output distribution comparisons require a running reference model and may not explain the cause of a discrepancy. Full evaluations are more expensive but can investigate task failures. A useful arrangement is continuous lightweight monitoring, tool-error and output-length statistics, and targeted evaluations when those signals change. Private customer tasks and progressively longer prompts extend coverage beyond public tests. Consistent harness parsing, official quantizations and published serving stacks should also become part of endpoint comparisons. Apply the same discipline to local engines.

Hybrid-model cache debugging benefits from three separate questions. Does the retained state satisfy each component's matching rules? Can capacity move between pools as context lengths change? Can the physical layout actually support the requested allocation? Liangsheng's unified radix tree, shared memory pool and compaction address these different layers. His correction is important: remapping concerns memory pages, with virtual page IDs remaining stable. A concrete validation should therefore check mappings and outputs after physical movement, and measure copying overhead rather than only counting free bytes.

Compiler search deserves an explicit investment boundary as well. Fix target hardware, representative input shapes, numerical tolerances and a search budget, then compare final runtime savings with compilation effort and expected reuse. Finally, treat the closing discussion of asynchronous rollout inside training as a direction for investigation. The supplied transcript does not establish reliable project names, synchronization boundaries or quantified gains. Before turning that mechanism into a capacity commitment, obtain the missing scheduling description and measurements.

Source

• Original episode: Why Inference Matters 12x More Than Training | YC Paper Club