What an Answer Key Can Reveal About an AI Scientist: Ian Quigley on Molecular Search

1. Guest Background

Ian Quigley joins this episode from Leash Bio to discuss work he participates in developing: environments for evaluating how AI agents reason about scientific experiments and choose their next actions. The supplied material does not identify his specific job title. His relevant perspective is that of a participant describing the environment's construction, the behaviors observed in its runs, and the questions the team still wants to investigate.

The introduction places that work in Leash Bio's screening program, describing DNA-encoded libraries that enable high-throughput screening of millions of compounds against thousands of proteins. The interview examines a second use for such experimental results: making them available as feedback inside an agent environment.

Quigley also recounts earlier internal work aimed at improving language models' handling of SMILES. As he explains it, a SMILES string records a traversal of a molecular graph in one dimension; different traversals can describe the same molecule. He recalls models struggling to convert between equivalent representations, followed by improvements that encouraged the team to pursue more substantial chemistry tasks. This is his account of the project's motivation, rather than a comprehensive measurement of when language models acquired chemical competence.

2. What the Episode Covers

The environment begins with a deliberate separation of knowledge. Researchers possess the outcomes across a premeasured screening landscape. An agent entering that landscape must choose which experiments to query under a limited budget. The answers already exist, but the agent does not receive the answer table. This makes it possible to study search under restricted information while retaining a reference for evaluating the resulting choices. The measured landscape is the relevant world here; it is not the entirety of possible chemistry.

The flagship task is a structure–activity relationship problem, or SAR. Given molecules that bind a target protein, can the agent find candidates that bind better? Quigley explicitly narrows the task to binding. Although broader SAR work can involve metabolism and other drug-like properties, those are outside the current exercise. A strong result therefore concerns the specified binding task, not the discovery of a finished drug.

His concrete example gives the agent a target, a catalog it can inspect, and an experimental budget of 100 molecules. Different agents develop different search policies and ways of allocating that budget. They articulate readable hypotheses about why molecular features might help or hurt, then use queries to test those hypotheses. The resulting record contains more than a final candidate: it connects proposed explanations, experimental selections, feedback, and subsequent decisions.

Quigley reports roughly 90,000 rollouts and says the evaluation separates agents across model families and capability levels. Those are descriptions of the team's work from the guest, rather than independently verified measurements supplied by the interview. The discussion also covers a preprint and blog release, together with plans for Harbor environment packages and many thousands of agent transcripts. Such materials would allow other researchers to inspect particular decisions and failure modes. The release discussion should nevertheless be read as the arrangements described during the conversation, rather than proof that every announced artifact was already available.

3. Core Views: Reasoning, Examples, and Limits

The answer key matters because it makes an outcome open to further questioning. Quigley offers a hypothetical agent that finds something only half as good as the best candidate in the environment. That is an illustration of evaluation, not a reported performance result. With the measured landscape available, researchers can investigate the gap: what opportunities did the agent miss, and what did it do instead? A respectable endpoint could conceal an inefficient path. Conversely, the path may reveal whether the difficulty lay in choosing a region, responding to feedback, or deciding when to stop. This counterfactual reference extends only across measured choices; it does not reveal the results of every molecule someone could invent.

The failure modes make the process argument concrete. Quigley describes agents that give up early, select molecules inconsistent with their own SAR hypotheses, or repeatedly test the same molecules despite a scarce budget. These should not automatically collapse into one diagnosis of weak chemistry. Early stopping raises a question about persistence. A mismatch between hypothesis and purchase raises a question about whether explanations guide action. Repeated queries raise a question about whether previous experimental information is being used effectively. The practical value of the trajectory is that it lets an evaluator distinguish these possibilities instead of treating every disappointing result alike.

The interview's clearest behavioral observation concerns leaving unproductive neighborhoods. According to Quigley, weaker models can keep making small changes around molecules that have already scored poorly. Stronger models abandon those areas faster. His point is more specific than saying good models move around more: they appear particularly willing to leave bad regions. Under a fixed budget, this distinction matters because continued local refinement consumes opportunities to investigate alternatives.

The mechanism remains unresolved. When asked why stronger models behave this way, Quigley says the team does not know. The observation supports a question about how agents respond to negative evidence; it does not establish a particular chemical reasoning algorithm. Nor does it justify an unconditional rule that earlier abandonment is always better. Whether persistence is worthwhile depends on what the remaining choices and feedback can support.

Plausible explanations present a related interpretive problem. Agents discuss whether a ring is helpful or a nitrogen atom is troublesome, producing narratives that sound like medicinal chemistry. Quigley says a limited amount of review by medicinal chemists found such thoughts plausible. That gives researchers a reason to examine the narratives, but it does not establish expert-level performance. The conversation does not provide a controlled human-versus-agent comparison with matched budgets and choices. An explanation's professional vocabulary also cannot establish that the explanation caused a successful selection.

Quigley is similarly careful about molecular understanding. He says decisions suffer when information about the molecules is removed, which supports the narrower conclusion that agents use that information. He then corrects his own language: weights are not changing during these runs. The agent is using information to make future choices within the task. That is distinct from demonstrating reinforcement-learning improvement or proving an internal grasp of molecular physics. The ablation is informative precisely when its conclusion stays close to what was changed and observed.

There is also an unresolved baseline question. Asked about a Bayesian optimization approach, Quigley judges the landscape too high-dimensional and spiky, but acknowledges that the team has not run that comparison. His judgment can motivate an experiment; it cannot substitute for one. The interview therefore does not establish superiority over that optimization baseline. A relative-performance claim would require an actual comparison under corresponding budgets and available actions.

Finally, premeasurement explains the environment's operational appeal. One collection of experimental results can support many different rollouts, amortizing the experimental investment and shortening feedback delays. Quigley qualifies the description of a zero-latency laboratory: it is close, rather than literally accurate. The speed comes from having conducted the experiments already. The evaluation remains meaningful because agents must still decide which hidden results to request. Together, these properties make repeated investigation of decision behavior feasible, while keeping the source of the feedback explicit.

4. Learning and Application

For a team adopting this approach, the immediate application is to make experimental-budget decisions inspectable. Quigley describes the environment as a way to measure agent mistakes, responses to physical feedback, and whether a new training recipe improves scientific decision-making. An evaluation workflow derived from that description would fix the catalog and budget before a run, then inspect hypotheses, selections, feedback, and strategy changes alongside the final score. This is a practical proposal drawn from the interview's design, not a protocol whose effectiveness the episode has already validated.

The available action set needs particular attention. Quigley notes that a medicinal chemist may want to add a methyl group or move a feature, only to discover that the corresponding molecule was never measured. The environment cannot return an experimental answer for an unavailable choice. For a human comparison, the permitted actions would therefore need to be presented meaningfully. Otherwise, a poor result could be difficult to separate from confusion about what the interface allows. For an agent evaluation, the same restriction defines the conclusion: success concerns choosing within this catalog, and does not by itself establish competence at unrestricted experimental design.

Resource planning also requires separating three kinds of numbers. Quigley reports approximately 1,000 proteins screened against a couple of chemical libraries, roughly two new environments produced per day at the time, and an ambition to reach 20 daily. These describe accumulated work, reported current output, and a future target. A research team planning its own evaluation should work from environments actually available to it, while treating the larger production figure as an aspiration. More environments would create additional opportunities for testing; their number alone would not settle whether an agent can handle unfamiliar biology.

Reinforcement learning is a possible next use, with an explicit condition attached. The team is considering other biological and scientific questions that could be reconstructed in this form, but Quigley says transfer of reinforcement-learning progress to new biology is not yet known. A useful follow-up evaluation would therefore record improvement inside the training environments separately from performance on genuinely new tasks. Fast, repeatable feedback makes training experiments easier to conduct. It does not remove the need to test what the resulting improvement applies to.

A broader application is to study the allocation problem Quigley identifies: when should a researcher persist, and when should the remaining budget go elsewhere? Different search policies could be examined through that question, provided their task conditions remain explicit. The episode's observation about leaving poor neighborhoods is a starting point for investigation, rather than a universal stopping rule. Quigley also says the team is incorporating more structural tool calls. Evaluators could compare otherwise corresponding runs with and without those tools, looking at decisions and results together with the resources consumed. The conversation reports the direction of development, not an established benefit from that proposed comparison.

Source

• Original episode: An Eval for AI Scientists That Knows the Answer Key - Ian Quigley, Leash Bio