Cloudflare Wants to Rewrite the Web’s Bargain for the Age of Bots

1. Guest Background

The guest is Matthew Prince, cofounder and CEO of Cloudflare, interviewed by Decoder host Nilay Patel in an episode titled “Can Cloudflare save the web from AI?” The conversation is not a generic discussion of artificial intelligence. It is an episode analysis of how one internet infrastructure company, positioned between websites and automated systems, interprets the collision among AI crawlers, open-web economics, Google’s search power, content licensing, and the internal structure of companies adopting AI.

Cloudflare is presented in the episode as an important infrastructure company that protects apps and services from bad actors, makes the internet faster, and sits between websites and AI tools or bots. That role matters because Cloudflare can give site owners some control over automated access: blocking bots, allowing them, or potentially allowing only paid access. Prince also broadens Cloudflare’s mission. Instead of describing it only as a service that makes the internet faster and safer, he says the company is trying to rebuild the internet as it should have been built if everyone had known from the beginning how important it would become.

Prince’s relevance comes from two forms of evidence in the episode. First, he runs a company that sees and controls a significant layer of internet traffic, so his claims about bot traffic, crawler incentives, site costs, and payment mechanisms come from Cloudflare’s infrastructure vantage point. Second, the interview turns to Cloudflare’s own use of AI inside the company, including layoffs of more than 1,100 people, about 20 percent of the company, and a management framework that distinguishes builders, sellers, and measurers. The episode therefore treats Prince both as a commentator on web economics and as an operator applying AI to his own organization.

2. What the Episode Covers

Prince begins with the advertising model. He says that for at least 30 years, the web’s growth has been driven mainly by advertising, and that Google was central to the first-generation web advertising ecosystem. Publishers, websites, forums, and communities tolerated search crawling because crawling returned traffic; traffic could then become advertising impressions, brand exposure, subscriptions, commerce, or some other economic benefit. It was imperfect, but it was a functioning give-to-get exchange.

AI breaks that exchange by changing who, or what, uses the web. Prince says Cloudflare first projected that automated traffic would exceed human traffic in the second half of 2027, then revised the estimate to the first half of 2027, before finding in May 2026 that automated traffic had already passed human traffic. He then adds a heavily qualified extrapolation: within five years, automated traffic could become 1,000 times human traffic. That number should be treated as Prince’s episode framing and forecast, not as a universally measured fact.

The key issue is not simply more visits. It is that the visitor no longer pays attention in the way the ad model requires. Prince argues that bots consume bandwidth, servers, and infrastructure, but do not click ads or respond to brand advertising. AI companies need talent, chips, and data; in his view, talent and chip scarcity may ease, while freely available open-web data is likely to become more restricted. That leads to his proposed distinction: content can remain free for humans, while bots pay because they do not participate in the old advertising exchange.

Cloudflare’s proposed mechanism is to put access control and price signals back into the infrastructure layer. Prince says Cloudflare is exploring HTTP 402 Payment Required with Coinbase, Stripe, and others so bot access can be charged at fractions of a penny per request. Basic access might cost as little as a thousandth of a penny to cover infrastructure, while publishers, academics, or other high-value information sources could charge a premium. His air-and-scuba-diving analogy captures the theory: markets require demand, but they also require constrained supply.

Patel keeps pressing on incentives. Even if micropayments work, why would creators still build websites? Why not go to TikTok or another platform? Prince concedes that public Wikipedia contribution data is down significantly, partly because users now consume Wikipedia-derived information through answer engines such as ChatGPT, Claude, and Grok rather than visiting Wikipedia itself. The open web as an information platform is under pressure; Cloudflare is trying to answer how creators can keep being rewarded if machine access becomes a dominant mode of consumption.

Google is the episode’s sharpest conflict. Prince argues that Google historically crawled to power search and send publishers traffic, but that using the same access for AI training without returning traffic changes the bargain. He says Cloudflare will, from September 15, default to blocking Google’s AI training crawl for free customers if Google cannot distinguish AI training from search crawling; site owners can turn that default off. Prince also says Google has committed to greater crawler transparency, potentially declaring crawler purpose at page access so publishers can allow some uses and refuse others.

Prince does not treat copyright litigation as the only path. He argues that technical identification, use declarations, and default controls may be more direct because large AI companies are easier to identify than state-backed hackers. The legal swirl remains important, though. Patel notes music companies suing Anthropic and White House support for OpenAI’s fair-use position in the New York Times case. Prince summarizes an Anthropic books ruling as saying training may be fair use, but stolen books are not. These are best read as episode claims by the host and guest, not as a full legal record.

The second half of the conversation turns inward to Cloudflare. Prince divides company roles into builders, sellers, and measurers. Builders include engineers and product managers who create products; sellers sell; measurers track organizational performance. He says most of the layoffs came from measurement functions, though not all measurement jobs disappeared. At the same time, he says AI has made builders roughly ten times as productive, increasing his incentive to hire engineers and product managers. Cloudflare began pushing companywide AI learning around mid-2025, including finance and legal rather than only engineering.

His concrete internal example is investor relations. Prince says Heather’s process for earnings-related investor documents went from about 20 people working for two weeks to about three minutes, and that investors found the documents better and less error-prone. Cloudflare also built Cloudflare OS to centralize company knowledge and records from systems such as Salesforce and Workday into a consistent interface, then open sourced it for customers. Structurally, Prince says the company wants to move toward roughly 12 direct reports per manager, compared with older norms around six, in order to flatten hierarchy. He stresses that the number is directional, not dogma.

3. Core Views: Reasoning, Examples, and Limits

Prince’s central claim is that the open web’s crisis is not just declining traffic; it is a change in the economics of traffic. In the old bargain, search crawlers took content and returned visitors. In the AI bargain, value may remain inside a model, chatbot, or agent, while the originating site still pays the bandwidth and server bill. His lunch-agent example makes the point concrete: if agent access has no marginal price, an agent may scan every local menu rather than a few relevant ones, while only one restaurant receives the eventual purchase.

The proposed answer is to make access priceable again. HTTP 402, use declarations, crawler identification, default blocking, and per-request payments all serve the same function: turning freely copied information into conditionally accessible data. The air-versus-scuba analogy is doing real argumentative work. Prince is not saying information should always be expensive; he is saying that when information is costly to produce, cheap to copy, and demanded at machine scale, a market may fail without some constraint on supply. Cloudflare’s position in front of more than 20 percent of the web makes it more than an observer; it can change defaults.

That is also the limitation. Cloudflare’s power problem does not disappear because the goal is economic rather than content moderation. The company says it serves customers and does not want to decide what is good or bad, but default settings can still reshape the relationship among Google, AI companies, publishers, and websites. A second limitation is that the creator incentive problem remains only partly answered: Patel’s question about creators moving to TikTok is real. A third limitation is evidentiary. Prince’s 1,000-times automated-traffic projection, Reddit-per-token comparison, and Park City newspaper licensing example are useful claims from the episode, but they should not be upgraded into universal measured facts.

Prince’s theory of content value is more provocative. He argues that AI markets will reward new, true, non-substitutable knowledge rather than repeated national political commentary or attention-economy rewrites. He describes LLMs as mathematical models of human knowledge with holes, and says AI companies want to fill those holes. The Park City newspaper example shows why local information could matter to a travel-planning AI; the Reddit example shows why community experience, long-tail questions, and real human discussion may be harder to substitute than a conventional newspaper article. The reasoning is not that Reddit is inherently higher quality than The New York Times. It is that scarcity and substitutability may be priced differently in AI licensing markets.

Patel’s counterpoint is essential: producing for models can change journalism itself. Hotel-room-level reviews, unpublished notes, extra photos, and metadata could give value to material that once fell on the cutting-room floor. But the same move could also shift journalism from a finished editorial product for human readers toward a fact-supply chain for model ingestion. Prince accepts much of that direction, but the conversation leaves unresolved questions about editorial judgment, source protection, reader trust, and the integrity of the work.

Inside companies, Prince applies the same cost-and-incentive lens. AI may first compress measurer work, not necessarily the most visibly creative jobs. Investor-relations documents going from two weeks to three minutes is his example of measurement and reporting work being rewritten by tools. If builders become more productive, engineers and product managers may become more valuable rather than less. The implication is uneven impact: AI does not replace all roles equally; it changes the leverage of different functions.

The episode also marks the boundary of data-driven management. Patel questions whether every industry resembles a software company where work naturally leaves logs in Slack, code repositories, and systems of record. Prince acknowledges that code commits and other single metrics can be gamed, and says the real test is whether behavior can be traced to revenue, cost reduction, or better collaboration. He also admits models sometimes misidentify stars and rejects the claim that Cloudflare has automated management. The strongest version of his view is that AI can expand managerial sight and surface overlooked talent. The dangerous version is an opaque, unaccountable promotion or firing machine.

4. Learning and Application

For publishers, knowledge communities, and content sites, the practical lesson is to separate human access, search indexing, AI training, and agent retrieval. Older crawler arrangements could often be treated as part of search distribution. That is no longer enough. A site needs to know who is arriving, why they are arriving, whether any traffic returns, whether costs are imposed, and whether the material has licensing value. Smaller sites may not be able to create a market immediately, but they can still inventory which assets are generic and which are local, specialized, community-based, or otherwise hard to replace.

If HTTP 402-style payments become viable, the useful application is not to charge for every page in the same way. Prince’s implied model is layered: keep human access free or supported by ads and subscriptions; charge a tiny machine-access fee to cover infrastructure; charge premiums for high-value reporting, research, databases, local intelligence, or distinctive expertise. The tradeoff is delicate. Too much friction may push AI companies elsewhere. Too little price may not sustain creation. Too much dependence on one infrastructure intermediary creates another concentration risk.

For media producers, the AI licensing market may reward information that models do not yet know but users will ask about. Local restaurants, hotel rooms, community experience, niche professional knowledge, original notes, and under-covered questions may become more valuable than another commentary piece on a national story. The boundary is editorial. Producing for model ingestion can dilute narrative craft, source protection, brand trust, and the human reader relationship. A healthier approach would separate human-facing editorial products from licensed machine-readable layers, with clear rules about what raw material can be shared.

For AI companies and platforms, the episode suggests that “can we crawl it?” is the wrong endpoint. The harder question is how creators receive both money and recognition after their work is absorbed into answer interfaces. Financial compensation might come through licensing, revenue shares, or per-access payments. Recognition is harder because answer engines can cut off visits, attribution, and status. Whether or not AI labs create the prize-like systems Prince imagines, model products need better source visibility, contribution signals, and feedback loops if they want knowledge communities to keep producing.

For business leaders, Cloudflare’s internal case is most applicable to measurement-heavy work: audit, reporting, document assembly, process tracking, investor materials, and management dashboards. Those workflows are digitized, repeatable, and easier to check. Tools can widen management span, flatten hierarchy, and let individual contributors coordinate agents. But the conditions matter: the company needs reliable data, defined outcomes, privacy boundaries, and human accountability. Organizations without digital work signals or outcome metrics cannot simply copy a software company’s management model.

Finally, AI performance detection should be treated as an assistive signal, not a verdict. It can help find people hidden by weak managers, low visibility, or organizational blind spots; it can also identify employees who need support. But code commits, response speed, camera observations, and other single signals can be misread or gamed. A healthier boundary is to let AI generate leads, require human review of context, connect evaluation to revenue, cost, collaboration, and customer outcomes, tell employees what data is being used, and keep promotion, layoff, and reward decisions explainable by accountable humans.

Source

• Original episode: Can Cloudflare save the web from AI? | Decoder