Security After the Warning Window Shrinks: Ami on Continuous Testing and Defensive Design

1. Guest Background

Ami is the guest in this installment of a16z / The Ben & Marc Show, indexed at 2,766 seconds, or approximately 46 minutes. The supplied guest background identifies him by first name but does not establish a surname, formal position, or organization. A responsible account therefore introduces him through the work he describes rather than attaching an inferred executive title or a biography the evidence cannot support.

That work spans vulnerability research, security product engineering, and observations from customer environments. Ami discusses an Atlas harness for vulnerability detection, his team's use of different models, and the behavior of customers confronting a PHP vulnerability. He also describes investigation and remediation capabilities in Wiz products. These references establish the practical perspective behind his answers: he is concerned with what researchers can uncover, what products can economically investigate, and what customers can actually repair.

The episode examines the widening mismatch between attack execution and defensive action. Its subject is broader than whether AI can find bugs. The interview asks what happens after a weakness is discovered, how a patch reaches a running system, and how security teams should spend limited attention. Ami's product and customer experience supplies concrete examples, but it also defines the evidentiary boundary: observations from those settings are informative without automatically becoming representative measurements of the entire security industry.

2. What the Episode Covers

Ami frames the discussion around what he calls “Black October”: a convergence of unusually consequential vulnerability discoveries and AI-enabled attacks. He argues that capabilities previously discussed as approaching threats are now being used against organizations. His most striking claim is that some attacks which once occupied elite attackers for weeks or months can now take around twenty minutes. The interview does not provide a common start point, sampling method, or benchmark. Twenty minutes is therefore an attributed observation, rather than a universal measurement of modern attacks.

AI development platforms provide his example of why smaller companies can attract attention. Ami says his team's blog initially listed nine platforms attacked in October and later listed fifteen. He connects the targeting to AI keys that attackers can exploit. In his account, a startup may discover an overnight intrusion only after secrets and customer keys have been taken. The useful implication is that possession of something valuable can matter more than company size. The supplied discussion does not establish the prevalence of this pattern across all startups or provide a complete incident dataset.

The PHP discussion exposes a different clock. Ami reports that almost all his customers had some PHP running and describes approximately fifty percent patching within the first two or three days. The denominator is unspecified: the figure cannot safely be converted into a precise share of affected assets secured. He also says some older versions were not receiving a patch. Rapid progress among systems that can be repaired can coexist with a substantial population for which routine patching is unavailable.

Availability is itself upstream of an enterprise's deployment process. Ami says the team worked on coordinated PHP preparations for months before information leaked prematurely. For Linux, he describes fixes taking weeks or months to reach all distributions. This separates several processes often collapsed into “time to patch”: vulnerability discovery, exploit preparation, production of an available fix, and deployment. Making the final internal step faster cannot eliminate a period when the organization has no suitable patch to install.

Customer expectations are moving beyond reports of those problems. According to Ami, buyers increasingly want security products to locate the relevant code, generate a fix and pull request, and carry out investigation, forensics, root-cause analysis, and attacker removal. The common objective is to reduce the manual handoffs between identifying risk and changing the situation. Yet he calls automated response frightening even while presenting it as a direction defenders must consider. The interview does not explain authorization, false-positive controls, or recovery procedures. It establishes pressure for a more complete defensive workflow, without demonstrating that unattended response is operationally settled.

3. Core Views: Reasoning, Examples, and Limits

The central analytical idea is that a security assessment can expire even when the application has not changed. Ami argues that a new model may discover an entry point its predecessor missed. His analogy is a door with a key hidden under the mat: a hiding place that appeared adequate becomes ineffective as the adversary acquires new abilities. Continuous testing follows from that changing threat capability. Its purpose is to revisit the assumptions behind an earlier assessment, rather than merely repeat a scan on a calendar.

An advantage in model access offers time, but not comprehensive coverage. Ami estimates that frontier models may lead open models by no more than three months in vulnerability discovery. He wants defenders to use that interval on critical infrastructure. The qualification matters: this is an estimate about a particular task, not a measured gap across every offensive capability or a guaranteed period of safety. He also acknowledges that there is too much code for defenders to examine everything. A stronger model cannot protect an area the research program never reaches.

This leads to a second argument: security spending needs an order of operations. Ami describes starting with deterministic exposure analysis, then examining unauthenticated internet-facing surfaces, followed by deeper penetration testing and code analysis. His examples range from minutes of attention per endpoint to ten hours for a critical application. Those durations illustrate differentiated effort, not fixed service standards. The underlying reasoning is that reachable entry points and consequential systems should influence where scarce investigation time goes.

His hardcoded-key example makes that priority tangible. Ami cites a mobile application whose embedded key exposed people's data. The example challenges an exclusive focus on difficult vulnerability discovery: an accessible credential can create an important attack path without requiring sophisticated exploitation. It does not establish that source analysis is dispensable. External testing and code investigation answer related questions about reachability and underlying causes. Indeed, both remain present in the layered approach he describes.

The same allocation problem appears inside model pipelines. Ami says model families, versions, and cyber-specific variants differ in capability, latency, and cost. His Blue Agent example begins with deterministic detection, progresses through cheaper models, and reserves more expensive reasoning for events that warrant escalation. This is an economic argument for selective investigation. It also leaves an evaluation question unanswered: the interview does not show how often an early stage fails to escalate something important. A pipeline that controls spending still needs evidence that its filtering preserves useful detection.

Scanning costs make the tradeoff explicit. Ami offers a five-million-dollar scanning example and says some customers spend tens of millions, while new code and new models create reasons to inspect again. These are his examples and spending claims, without supporting accounts or comparative returns in the supplied material. He expressly says the broader budget problem remains unsolved. Continuous security therefore cannot simply mean buying the most expensive complete scan whenever a stronger model appears. It requires choices among external testing, internal analysis, disk inspection, and runtime observation.

Ami's architectural argument addresses what happens when those choices fail to find a weakness in time. Sandboxes, isolation, and restrictions at the network, process, and image levels assume that a vulnerability will eventually exist. Purchasing a tool does not establish those boundaries. His criticism of reporting and configuration checks similarly asks whether work improves readiness for an actual attack path. It should not be expanded into evidence that every compliance activity lacks value, just as his forceful remarks about container escape do not establish that every container is universally penetrable.

Finally, the episode's forecasts deserve a separate evidentiary status. An interviewer anticipates frequent unauthenticated remote vulnerabilities and possibly a return of internet worms; Ami asserts that AI attacks outperform humans. The supplied discussion contains no systematic comparison establishing the latter claim and cannot establish the frequency of future events. The operational argument survives that uncertainty: repeatedly test important assumptions, allocate effort deliberately, and limit the consequences of failure. The strongest predictions remain predictions.

4. Learning and Application

For a new company, the most direct application is to make one remediation path work from beginning to end. Ami's proposed chain includes ticket intake, development, testing, CI/CD, and operations. A practical implementation derived from that argument would start with a service whose ownership and testing are clear, then examine whether each handoff has usable inputs and a completion condition. A generated patch is an intermediate artifact. The defensive benefit depends on validating and delivering the change. This approach suits a team still designing its development process; it does not imply that every existing system can immediately support automatic repair.

Legacy organizations face a different first task. Ami describes old software, thousands of developers, unclear ownership, and knowledge that has never been written down. Before expanding agent execution, a team can identify who owns an affected resource, how a change can be tested, and who can decide to deploy it. These are practical deductions from his diagnosis rather than a deployment specification supplied in the interview. The tradeoff is slower initial preparation. Its purpose is to prevent automation from reaching a decision point for which neither context nor an accountable person exists.

Red–blue coordination provides a bridge when repair cannot happen immediately. Ami describes external adversarial testing informed by internal context, with findings passed to Blue Agent when engineering considers a fix too expensive. In practice, a team could keep the remediation task, interim detection, and responsible owner attached to the same finding. That preserves the distinction between observing an attack and removing its entry point. Detection creates an opportunity to act; it does not guarantee timely intervention. Without an actionable response process, connecting the agents may simply create another queue.

Teams holding AI keys have a narrower measure available: Ami recommends usage limits to constrain charges after theft. Limits should reflect legitimate demand. An excessively restrictive allowance can interrupt normal use, while a generous one leaves more room for abuse. This measure addresses a particular financial consequence after a credential is obtained. It neither prevents the intrusion nor reverses a disclosure of customer information. Its value is clearest when treated as a bounded loss-control mechanism.

Ami's recommendation to remove unprepared systems from the internet can also become a concrete review. For each exposed service, ask why direct external access is necessary, which entry points could be narrowed, and what changing access would mean for users and operations. Reducing reachability may reduce opportunity for attack while imposing delivery or maintenance costs. The decision therefore belongs at the level of the particular system, rather than becoming an indiscriminate instruction to disconnect everything.

Public services illustrate the remaining distance between discovery and sustained defense. Ami describes Scan for Good as helping institutions find problems and encouraging continuing red scanning, while acknowledging that it cannot solve all their issues. A scan can establish a starting point for a resource-constrained organization. Someone must still interpret the findings, arrange changes, and maintain follow-up. The useful procurement question is consequently broader than coverage: who receives each finding, what happens when repair is unavailable, and which responsibilities remain with the institution? The episode's practical value lies in making those unfinished dependencies visible.

Source

• Original episode: Cyberattacks Used to Take Months. AI Can Do It in Minutes.