When Building Gets Easier, Product Judgment Has to Get Better

1. Guest Background

Tulsee Doshi leads product for Google's frontier models, including Gemini. In this conversation with Lenny Rachitsky on Lenny's Podcast, she examines what product management becomes when the underlying technology keeps changing and its eventual capabilities cannot be fully specified in advance. Her perspective connects model development with the choices application teams make about quality, interaction, and delivery.

The breadth of her portfolio matters to that discussion. Alongside Gemini, she describes responsibility for image and video generation and editing, including Nano Banana; the music model Lyria; the open-model family Gemma; and audio capabilities spanning dialogue, understanding, text to speech, and translation. Quality in this setting includes visual representation, vocal delivery, and conversational behavior as well as the substance of an answer.

Doshi also says she spent six years working in responsible AI. Her account of evaluating skin-tone representation gives the episode a concrete example of how a measurement framework can shape product decisions. Earlier, at Stanford, she researched flash teams with Michael Bernstein: assembling crowdsourced human workers around complementary strengths. She sees principles worth exploring in systems of subagents, while explicitly distinguishing the original human-team research from the host's framing about agent swarms. These experiences support her perspective on evaluation and coordination; they do not establish that lessons transfer automatically between people and agents.

2. What the Episode Covers

The interview begins with the tension between intentional model development and discovery. Doshi says teams should define desired capabilities, user outcomes, and evaluation goals. Researchers monitor training runs, track metrics, and intervene. Yet those activities do not make the final model behave like a conventional software feature whose outcome can be specified completely beforehand. A PM must inspect what emerges, identify unexpected strengths, and work out where those strengths could produce a compelling experience. Product work therefore continues on both sides of training: shaping the intended direction and learning from the actual result.

Doshi places that work within what she describes as Google's full-stack strategy, stretching from hardware through models to end-user products. In her account, competing at the frontier coexists with serving different points on a cost-quality curve. Someone searching for information, an engineer building software, and a defender protecting infrastructure present different needs. A portfolio can address those differences rather than treating one model choice as appropriate everywhere.

Some surrounding claims require careful attribution. Doshi expresses hope that listeners will be using Gemini 4 Argon when the conversation appears and assesses it as a frontier model. Those statements establish her expectation and judgment, not its independently verified release status or comparative performance. She also explicitly says she should check the figure when mentioning fourteen products with more than a billion users. The strategic argument does not depend on treating that tentative count as settled.

On safety, Doshi describes safeguards extending from training into deployment, including monitoring model chain of thought for deviations from directed actions and intervening. Her optimism rests on sustained scrutiny and deliberate mitigation. She also cites Wiz's Scan for Good program, saying it helped a hospital system identify and patch a vulnerability. The interview offers that as an illustration of defensive use, without independent case documentation or a quantified outcome. It does not demonstrate that monitoring or defensive applications resolve the broader safety problem.

The conversation then moves into the product experience. Flow and NotebookLM illustrate experimentation with different forms of interaction. Doshi contrasts an accessible search AI Mode with Antigravity, where users familiar with terminals, IDEs, and slash commands can work with more complex controls. The design question is which interaction suits the audience and task.

Feedback connects these choices to model investment. Doshi describes working with internal customers such as Search, Gemini app, YouTube, and Labs, alongside external customers, to understand what would meaningfully change their experience. That creates a route from downstream needs back to capability development. The remainder of the episode explores the PM skills and organizational practices needed to make that route useful.

3. Core Views: Reasoning, Examples, and Limits

Rachitsky frames the renewed value of PMs around a problem created by easier implementation: people can build more things that nobody wants or that do not work properly. Doshi agrees, emphasizing taste, insight, and consensus. The reasoning is that faster execution leaves the selection problem intact. Someone still has to identify a worthwhile need, define a good result, and align contributors. Without that structure, speed can amplify confusion. This is their explanation of product management's value; the episode does not supply labor-market evidence proving a general resurgence in demand for PMs.

For Doshi, this judgment reaches into the model itself. Transparency, communication, and follow-up questions are elements of UX. She describes collaboration between PMs and researchers that turns user feedback and desired behavior into specifications, evaluation rubrics, training data, and reward modeling. The implication is that an aspiration such as helpfulness needs observable criteria before it can guide development. A team might need to distinguish when a clarification is useful from when it interrupts progress. That distinction is an application of her approach, rather than a specific training rule disclosed in the interview.

Her hiring criteria connect intuition with technical participation. She wants research PMs who use models daily, have their own stress tests, inspect data, build evaluations, and revise plans as conditions change. Personally meaningful tasks provide useful sensitivity: someone familiar with creative writing, animation, synthesis, or difficult financial analysis can notice differences that a casual trial misses. But familiarity also gives the observer a particular perspective. Doshi's approach makes that perspective a starting point for investigation, with structured evaluation providing a way to examine it. She acknowledges that evaluation capability remains uneven within her own team.

Her recommendation to begin with five prompts and grow toward roughly twenty is best understood in that context. The prompts should expose differences between models and force the evaluator to explain what counts as good. For a long, multi-step, multi-turn task, defining success may be harder than writing the prompt. A small collection can reveal ambiguity in the standard before a team scales its testing. Neither five nor twenty establishes statistical reliability or broad user coverage; the episode offers an accessible entry point, not a validation threshold.

The skin-tone example shows why an evaluation can fail before any model receives a score. According to Doshi, the six-tone Fitzpatrick scale had a healthcare-related purpose and skewed toward lighter skin, making it poorly suited to global representation. Her team incorporated Professor Monk's ten-tone scale into evaluations and work on Pixel cameras and image generation. The analytical lesson concerns fitness for purpose: a familiar instrument can obscure differences relevant to the product. She reports no quantitative gain, so changing the scale cannot be translated into a numerical improvement claim.

Evaluation also needs to reach the delivered experience. Doshi identifies judge quality, production filters and mitigations, and serving as factors that affect what users encounter. Audio makes the distinction tangible: the content can be satisfactory while naturalness, vocal character, or inflection suffers. An isolated model test consequently leaves open questions about production quality. Equally, a disappointing production result needs investigation across the chain before the underlying model receives all the blame.

Her adjustable 60/20/20 framework addresses uncertainty in product planning. It combines capabilities that work today, capabilities that work imperfectly but appear close, and promising capabilities that could unlock a stronger experience. The proportions are a planning heuristic, not a measured law. Doshi explicitly says she cannot precisely predict model capabilities three months ahead, despite working on their development. A defensible bet therefore needs specific signs of progress and identifiable gaps within the product.

Finally, the harness must remain open to examination. Doshi includes instructions, tools, skills, context, and accessible data in that surrounding environment. Structures designed for an older model can constrain a stronger one. Rachitsky relays Boris Cherny's suggestion to occasionally remove skills and observe the result; Doshi emphasizes repeated testing. This supports checking the contribution of existing scaffolding. It does not establish that every instruction or tool should disappear as models improve.

4. Learning and Application

A practical starting point is one recurring task whose quality you can judge. Following Doshi's advice, create a few discriminating prompts, retain the outputs, and write down what makes a response successful or unusable. Someone testing synthesis might examine whether important relationships survive; someone testing creative writing might examine whether the result achieves the intended effect. These are proposed applications, not experiments reported in the episode. Expand the examples as new failures appear, and check whether your criteria represent the intended users. For longer workflows, assess the complete task and the delivered experience, including the evaluator and production setup.

For planning, sort capabilities into working now, nearly usable, and promising but unresolved. The useful part of 60/20/20 is the evidence behind those categories. Record what the current prototype delivers, which gap blocks the next improvement, and what observation would justify further investment. A dependency on future capability needs an explicit boundary on waiting and spending. After a model upgrade, replay retained tasks against the existing harness and selected simpler alternatives. Removing a skill is informative only if the comparison shows what happens to task quality.

Team development requires room for this work. Doshi describes useful technical skills being shared through a repository and colleagues teaching others to inspect data and long traces. A concrete adaptation is to build a reusable workflow around an actual task, then show peers its inputs, failures, and corrections before encouraging adoption. Her planned two-day November workshop remains a plan in this evidence. Likewise, her tenfold acceleration claim has no supporting measurement. A team should examine the time saved and the checking work added rather than adopting that multiplier as a target or forecast.

The second occurrence of a task can serve as a reminder to consider automation. Doshi preserves exceptions such as strategy writing, which makes this a trigger for judgment rather than a delegation rule. Work with clear inputs, a stable output structure, and inspectable results is a sensible candidate. A repeated strategic question may still require fresh human reasoning. The decision should turn on what can be checked and what responsibility the person needs to retain.

Her two-week post-launch report offers a specific division of labor. A model can assemble usage patterns, dashboard metrics, feedback, and social discussion. The PM then explains which signals matter, what uncertainties remain, and what decision follows. Source checking remains necessary: assembling material does not establish that a conclusion is justified. A prototype, evaluation, or relevant data can give the resulting proposal substance that collaborators can inspect. This is the practical meaning of Doshi's argument that PMs should deliver more than written requirements.

Finally, track sustainability alongside output. Doshi acknowledges long hours and an unresolved management problem, while hoping AI can remove unwanted work and churn. Managers can separately examine hours, repeated rework, frustrating tasks, and the work individuals find meaningful. Motivation differs across people. Increased throughput or enthusiasm about a tool does not demonstrate reduced burnout; the interview's own account leaves that outcome open.

Source

• Original episode: Why PMs are suddenly valuable again | Tulsee Doshi (Head of Product, Gemini)