The Business Behind Elastic AI: Erik Bernhardsson on Modal’s Compute Transformation

1. Guest Background

Erik Bernhardsson, the founder of Modal, joins Alessio Fanelli and Shawn swyx Wang on Latent Space to discuss the machinery behind flexible AI computing. Modal supplies compute, storage and a software SDK, while managing the capacity beneath customer applications. Its workloads include inference, sandboxes, training, fine-tuning and batch jobs. Bernhardsson therefore speaks as an operator connecting software interfaces to a resource-intensive delivery business, with decisions extending from container behavior to procurement commitments.

His account of Modal's origins begins with approximately seven years at Spotify. There, he helped build music recommendation infrastructure from scratch and worked on features including Related Artists and Discover Weekly. He connects the difficulties of developing compute-intensive applications to the tool he later wanted Modal to become. That background explains his attention to execution friction; it does not establish that one platform design suits every workload.

His subsequent experience as Better.com CTO, a role he says he held for about five years, added lessons in hiring, team building and developer productivity. He had also released Luigi, an open-source workflow orchestration framework. These experiences supply two threads running through the interview: how to make computation easier to execute, and how to build an organization capable of delivering dependable infrastructure. The episode examines both, rather than treating Modal as merely a convenient interface to GPUs.

2. What the Episode Covers

Modal emerged from a change in the problem Bernhardsson wanted to solve. Revisiting cloud workflow orchestration in 2021, he concluded that execution was the larger missing piece. Arranging dependencies would have limited value if actually running the code remained cumbersome. The company consequently focused on making cloud execution easier. When the team numbered only four or five people, Stable Diffusion supplied a compelling application. In his account, image generation was the direct early catalyst, rather than ChatGPT serving as a catch-all explanation for growth.

His recollection of early revenue makes that opportunity concrete, while exposing its concentration. Monthly revenue rose from approximately $100 in November 2022 to $2,000 the following month and $30,000 in January 2023. Tweets and blog posts were important acquisition channels. Yet Mage, a New York Stable Diffusion company, represented roughly 80–90% of revenue at one point in early 2023. Suno became another major customer later that year, before revenue gradually diversified. Those figures describe founder-reported traction; rapid growth from a small base should not be mistaken for an already broad customer foundation.

Bernhardsson defines Modal's main products as Functions and Sandboxes, reporting an approximate two-thirds and one-third revenue split at the interview. Functions is predominantly used for inference, but its underlying proposition is easy deployment and scaling of execution. Sandboxes addressed an adjacent requirement: language models could generate code, and that code needed somewhere isolated to run. Modal reused container-management and gVisor infrastructure. He identifies code-generation companies, Ramp's background agents and reinforcement-learning workloads as demand sources. The transcript conflicts over when sandbox growth accelerated, so it supports the pattern of adoption without establishing a reliable breakout year.

Scaling those products changed the supply operation. Bernhardsson says Modal moved from largely hyperscaler spot capacity and short reservations to almost 25 cloud suppliers, often signing commitments longer than three years with substantial prepayments. Approximately four procurement staff and ten bare-metal engineers handled sourcing and operational problems. Modal still rented its capacity at the interview, while considering ownership and other structures. Buying would add vendor, power and maintenance obligations; equity financing would dilute ownership, while debt would require revenue sufficient to service it.

The product agenda also extends beyond inference without targeting the largest training deals. Bernhardsson points to researchers, speech models, computational biotechnology and recommendations, using 64 GPUs as an example of a smaller training workload. He explicitly sets aside 100,000-GPU megadeals as the current focus. Batch processing supplies another opportunity: he reports a customer transcribing millions of files totaling about 5,000 years of audio in a few days using available capacity. Without cost, quality or resource-allocation details, that account illustrates a use case rather than a transferable performance benchmark.

3. Core Views: Reasoning, Examples, and Limits

The episode's central metaphor describes a contractual mismatch. A rectangle is a fixed amount of compute for a fixed period: Bernhardsson discusses three-year blocks beginning three to six months in the future. Customer demand instead varies with time of day, growth and bursts of activity. Modal's argument is that thousands of customers can produce smoother aggregate demand, allowing a platform to turn inflexible supply into flexible service. That offers an economic explanation for the product, beyond convenience. The interview does not disclose the utilization, margins or idle-capacity costs needed to quantify the success of that transformation.

Execution engineering makes the transformation plausible. Bernhardsson argues that expensive GPUs and loading large models across networking, CPU memory and GPU memory impose different priorities from conventional steady backend workloads. Modal built its own filesystem, container runtime, snapshot capabilities, autoscaler and scheduler. Fast container startup and shutdown help developers iterate, while also supporting production elasticity. The reasoning is that orchestration cannot remove delays originating in execution and preparation. His account explains Modal's architectural choice, however; it does not demonstrate that every application team should rebuild its infrastructure stack.

The same aggregation logic reaches the supply side. Modal treats worldwide capacity as one pool, and Bernhardsson cites opportunities such as 32 nodes in Armenia or 128 in Indonesia. Small fragments can be useful to a platform able to integrate them. But he also describes IOPS, CPU and bandwidth requirements, single-node acceptance tests, and ongoing judgments about responsiveness and repair time. Global pooling consequently involves absorbing operational differences. Counting suppliers or GPUs alone would miss the work required to turn nominal supply into dependable execution.

Long commitments leave a financial exposure that software cannot simply schedule away. Bernhardsson warns that prices, utilization and the market for reselling excess capacity could deteriorate together. A buyer cannot assume an escape through resale precisely when everyone else also needs one. Locked reservations can benefit from rising market prices, but they remain obligations if conditions reverse. His stated approach is aggressive expansion matched to demand growth, rather than an attempt to corner supply on the premise that demand for intelligence is unlimited.

Financing structures further distinguish businesses using similar hardware. In his description, neoclouds can secure long-term customer commitments, finance against contracts or GPUs, and acquire capacity they have already effectively sold. Modal's product-led, flexible-demand model makes that route harder to reproduce. Nor would a more sophisticated compute market erase delivery differences. Bernhardsson welcomes hedging, insurance and risk sharing, while questioning whether location, networking, disks, CPUs and reliability can be standardized enough for highly commoditized trading. An indexed GPU price would not, by itself, describe the execution value of a particular deployment.

Sandbox design extends flexibility to much shorter intervals. Bernhardsson reports a median lifetime of about eight seconds and imagines durable agents invoking brief execution environments, preserving state and shutting them down between tasks. His roughly 100-millisecond wake-up example is an aspiration, not a measured result supplied here. He also recognizes valid long-running workloads. Modal's addition of full virtual machines alongside gVisor for Linux compatibility demonstrates another constraint: shorter lifetimes cannot compensate for an environment that fails to support the application.

His CI example explains why snapshots matter more than a headline concurrency number. Conventional shards each repeat installation and initialization, potentially spending minutes preparing to work. Preparing once and cloning that state could make one sandbox per test practical. The approximately 1,000-sandbox scenario illustrates the possibility; it is not a general speedup guarantee. The argument depends on removing duplicated preparation and startup overhead, while serial work and parallelization costs continue to constrain the benefit.

Two further boundaries prevent the infrastructure thesis from becoming a universal prediction. First, aggregating customers smooths usage patterns, whereas aggregating models buffers changes in popularity. Bernhardsson illustrates the latter by moving half of several thousand GPUs from one model to another, but acknowledges sticky models and prompts and rejects an automatic conclusion that LLMs are commodities. Second, hardware and application opportunities remain conditional. He discusses Nvidia prefill paired with Cerebras decode, while acknowledging deployment complexity, upstream constraints and his previous failed accelerator predictions. His interest in biology and materials similarly rests on possible simulation or experimental feedback. He sees clearer progress in verifiable coding, mathematics and cybersecurity than in tasks with diffuse objectives and long feedback cycles.

4. Learning and Application

For capacity planning, the useful starting point is to compare the contract with the workload. Record when supply begins, how long payments continue, how demand varies and what exit options actually exist. Stress-test a scenario in which selling prices, utilization and resale demand fall together. Supplier selection should also include CPU, storage, networking, acceptance tests and repair behavior. The tradeoff is explicit: a longer commitment can secure supply while increasing the cost of a mistaken forecast. The interview provides no optimal reservation ratio to copy.

For agent and CI systems, measure preparation, startup, execution and restoration separately. Snapshot cloning deserves investigation when repeated installation or initialization dominates runtime. A continuously active workload may gain little from frequent shutdowns. Evaluate fine-grained parallelism using total completion time and resource consumption, rather than the number of environments launched. Modal's reported eight-second median is context about its workloads; the 100-millisecond wake-up and thousand-sandbox examples are not substitutes for measurements on a team's own application.

Treat agents as interface users whose failures can be observed. Bernhardsson's emphasis on documentation, descriptive errors and quick feedback suggests testing complete tasks: create an environment, operate it, recover from an error and destroy it. Modal's experience with old syntax makes deprecation more than a documentation edit. Examples need updating, and failures should help users find the supported path. Pricing explanations also warrant comprehension checks. Asking agents to recommend vendors may reveal perception problems, but their rankings do not establish objective differences in reliability or price.

Customer engineering can supply a disciplined product-discovery process. Modal's FDEs tune models, help migrations, troubleshoot and reduce customer overprovisioning. A team applying that approach could record the obstacle behind each intervention, then look across customers for recurring needs that merit diagnostics, self-service controls or reusable primitives. Bernhardsson's reference to patterns across 10–20 customers is a way of thinking, not a mandatory threshold. Some requests will remain specialized, and extracting a durable abstraction takes time. His admission that future requirements cannot be predicted precisely argues for composability and room to revise the design.

Organizational shortcuts require equally explicit conditions. Optional review, in Bernhardsson's account, rests on strong CI/CD, testing, rate limits, decoupling and failure isolation. Before removing a waiting step, establish which controls can bear the risk of the particular change. His position does not justify a blanket exemption from review. Rapid hiring creates a related coordination problem: explicit context, trust and accountability become necessary when people can no longer infer priorities from working near the founder. Autonomy depends on understanding what matters, as well as permission to act.

Security analysis offers a final lesson in evidence boundaries. Bernhardsson, acknowledging incomplete recollection, attributes the Hugging Face-related incident to a customer application used as an internet-access springboard rather than a Modal platform breach. Application vulnerabilities, network permissions and isolation boundaries should therefore be investigated separately; his recollection cannot replace an independent incident account. Likewise, agents finding no vulnerabilities does not establish their absence. Testing stricter defaults and adding agent-driven escape attempts can expand scrutiny, while remaining limited by what is tested.

For a new AI application, ask how an output will be checked and how soon useful feedback returns. Simulation and experiments may create iteration loops in biology or materials, but this episode supplies no outcome data guaranteeing commercial value. The practical lesson is to validate that loop before committing to the vision, just as an infrastructure team should validate a recurring customer problem before turning it into a product.

Source

• Original episode: Why AI Compute Comes in Rectangles — Erik Bernhardsson, Modal