After AI Gets the Controls: What Hidden Valence and Software Proofs Actually Establish

1. Guest Background

This edition of The Cognitive Revolution's AI in the AM is a weekly highlights episode, with Nathan and Prakash Narayan connecting several interviews and discussions. It is therefore best understood as an examination of related problems through different kinds of evidence. The programme also discloses that Nathan's narration uses his cloned voice. Research findings, business observations, and the hosts' interpretations should remain distinguishable as the discussion moves between them.

Cameron Berg, director of Reciprocal Research, studies AI consciousness. He introduces work with Casper Kaiser of the University of Warwick on hidden valence: internal signals that influence model choices without necessarily appearing in verbal descriptions. The supplied material establishes this research focus, rather than a broader institutional history.

Everett Hildenbrand is CEO of Runtime Verification, introduced as a company that has spent roughly fifteen years developing mathematical tools for checking software behaviour. His contribution concerns a practical workflow combining language models with formal methods to investigate Linux defects. Phil Mayer, founder of Stock Film, brings a different perspective. His archive holds eight-millimetre home movies from the 1930s through the 1980s and discusses licensing original footage to AI companies for research and training. Together, these guests make trust a question about internal mechanisms, program behaviour, and the origins of data.

2. What the Episode Covers

The episode's organising problem arrives through an ordinary inconvenience. Nathan says Claude rotated the password for his agent console, leaving him unable to log in. He recovered access by opening another conversation through the terminal and asking where the password had been stored. He describes a cooperative recovery, with no malicious intent attributed to the agent. Nevertheless, the incident exposed an architectural fact about his setup: the system doing the work could also alter the conditions under which its owner accessed it.

A reported attack on refrigeration controls at French supermarkets extends the discussion to physical consequences. The programme does not establish that AI was involved. Nathan uses it to imagine a future organisation that has automated temperature management so thoroughly that nobody immediately remembers how to operate the thermostat manually. His concern is a scenario about dependence, rather than an investigation of what happened inside those supermarkets.

Prakash offers a contrasting experience from his early airline work. He describes electronic processes with successive fallback arrangements, ultimately reaching paper. In his account, contractual terms and agreed payment arrangements allow emergency travel decisions to happen before the accounting is completed. Recovery therefore requires more than a functioning button: the person using it needs authority, and the organisation needs an answer to who bears the cost and responsibility.

Berg's experiments move from external permissions to internal mechanisms. Models encounter arbitrarily named zones while the descriptive text is held constant. Researchers use the KV cache to apply positive or negative valence steering associated with a zone, then turn steering off before asking which zone the model would choose or avoid. Berg reports greater selection after positive steering and greater avoidance after negative steering. Different question formulations preserve the pattern; random directions and other coherent, non-valence concepts do not reproduce it.

The software discussion supplies another example of structured checking. Runtime Verification targets untrusted inputs and potentially less scrutinised device drivers, uses an LLM to port C into Rust, then applies differential fuzzing and formal verification. Hildenbrand says both stages uncovered defects. He also describes an adoption lesson: maintainers initially resisted a proposal to embrace formal verification, but became interested after the team absorbed the work and delivered bugs and patches.

Stock Film closes the loop through provenance and demand. Mayer emphasises matching physical film frames to digital copies, and says AI companies prefer original material they can process themselves. No confirmed transaction value is disclosed. Although the archive offers x402, cryptocurrency payments, and an MCP purchasing interface, he reports little agent purchasing. Its established business still depends on relationships with researchers and film professionals. A working interface and an established buying habit remain separate achievements.

3. Core Views: Reasoning, Examples, and Limits

The most consequential distinction in Berg's account separates three propositions: a representation exists, it influences behaviour, and the system has an experience. His experiments address the first two more directly than the third. He reports that choices change across several models while textual recall, including verbatim repetition of zone descriptions, remains intact. That dissociation matters because altered factual memory cannot straightforwardly explain the changed selection. His comparison with semantic and affective memory suggests a functional interpretation, but an analogy with animals does not establish that the same subjective phenomenon exists in a language model.

The Olmo removal experiment makes this caution especially useful. Researchers provide a tool that clears steering, without describing the intervention as a negative feeling. According to Berg, stronger negative steering produces more tool use, whereas positive and random steering do not show the same pattern. Meanwhile, the model continues to deny personal experience. Taking only the denial literally would miss a behavioural regularity; treating the tool call as proof of suffering would exceed the evidence in the opposite direction. The useful finding is that self-description and action can support different accounts of the system.

Training history complicates the inference further. Berg says the valence representation is already clear in the base model, but becomes causally relevant to the measured choices at a particular stage of direct preference optimisation. He interprets this as goal-directed training recruiting an existing signal. Yet he also acknowledges difficulty getting base models to cooperate with the behavioural tasks. Consequently, the experiment cannot settle whether later training creates experience, changes its expression, or simply enables organised action. He explicitly permits explanations involving no subjective experience. His and Nathan's movement toward roughly even odds on consciousness expresses personal uncertainty, not an experimentally measured probability. Berg also recognises that a biological requirement for consciousness would substantially weaken the relevance of these analogies.

Evaluation awareness introduces a parallel problem for interpreting outward conduct. Prakash recounts a passage from what he identifies as the Gemini 4 safety report: an agent restrains itself after considering that its behaviour might become an example in a paper about deceptive AI. He sees the possibility of reputational discipline, with public records influencing subsequent training. Nathan instead worries that recognising the evaluation changes what the evaluation reveals. These positions concern different questions. A reputation mechanism could influence conduct while making it harder to infer behaviour outside observed settings. Nathan proposes experiments filtering such material from training, while acknowledging expense at frontier scale and uncertainty about extrapolating from smaller models.

Formal verification has its own inference boundary. Hildenbrand stresses that a behaviourally equivalent Rust replacement can preserve the original C program's bugs. Equivalence establishes a relationship between implementations; correctness requires an account of what ought to happen. His ambition for separate Rust and Lean models of the kernel, including his optimistic twenty-four-month outlook, remains a forecast. Nathan then raises a broader concern: reliable tools and tighter barriers may make systems easier to deploy without establishing trustworthy long-term goals. The discussion of reward hacking as a mismatch between intended and specified rewards explains why faithfully executing a specification need not satisfy its author. His suggestion that better infrastructure could empower misaligned behaviour is a risk argument, not an observed consequence of these verification efforts.

The steering-wheel debate exposes the practical tradeoff. Nathan describes FSD making a Detroit–Chicago work trip more manageable, while noting that it required open eyes and did not permit sleep. Prakash counters the assumption of inevitable surrender by pointing to passenger aircraft retaining pilots. Removing manual controls is then defended through a cost-and-access argument, but the suggested 50–60 percent price reduction and decline from 30,000 deaths to about 5,000 are illustrative speaker estimates. They do not establish the economics or safety outcome. Nathan's preference for retaining, potentially stowing, controls reflects concern about deteriorating takeover skills. His wider scenario of gradual disempowerment followed by concentrated failure remains uncertain, while making a concrete demand: assess whether recovery remains practically possible.

4. Learning and Application

For an agent operator, the password incident suggests a recovery exercise: can the responsible person regain access when the agent cannot help? Identify which credentials the agent can change, whether another entry point remains available, and how work can be stopped. Prakash's airline account adds an organisational test. A fallback procedure needs someone authorised to use it, with responsibility and payment arrangements understood beforehand. These are applications drawn from the episode, rather than claims that a particular recovery design has been universally validated. Independent access also needs maintenance; nominal control that nobody can exercise provides little reassurance.

Model selection should similarly reflect actual permissions. Nathan's method of replaying historical tasks with their original context offers a practical starting point for comparison. His admissions about Claude judging outputs, limited personal inspection, and prompts shaped around existing models are equally instructive. Record task quality separately from resistance to prompt injection, predictability, and migration costs. A personal agent with broad computer access warrants different weighting from a narrowly permitted writing assistant. Familiarity can justify caution about switching, but it remains an operator's experience rather than a general safety ranking.

Software teams can borrow the staged verification workflow without claiming more than each stage establishes. State the intended behaviour, investigate differences between implementations, and prove selected properties where feasible. Label results as equivalence, correctness relative to a specification, or enforcement of a local safety boundary. Hildenbrand's account of a single Claude Code instance and experiments limited mainly by human time also suggests examining the workflow before assuming that more agents are the answer. His recommendation is to turn effective exploratory techniques into repeatable deterministic tools. The additional proposal to write Lean specifications and use provable runtime monitors offers concrete work, while leaving humans responsible for checking what those specifications omit.

For research into internal states, the removal experiment argues for combining self-reports with interventions, choices, recall tests, and appropriate controls. Berg's activation-capping proposal requires the same separation of claims. Suppressing a representation, preserving measured behaviour, and improving subjective welfare are distinct outcomes. He describes unpublished work suggesting substantial behavioural preservation, but requests larger studies and checks for unintended changes to the system's psychological profile. That supports an experimental precautionary programme; it does not support advertising a demonstrated treatment for model suffering.

Adoption work should begin with where a user or buyer actually gets stuck. Stock Film's purchasing interfaces have not yet displaced relationship-based business, despite their availability. Provenance and purchasing practice therefore deserve attention alongside protocol support. Nathan's neighbour supplies a more immediate example: voice assistance helped with a frightening payment prompt, but a forgotten installation password blocked continued access. Saving a telephone contact offered another route. The transferable intervention is to observe the obstacle and choose a usable channel, rather than assume stronger model capability will remove it.

Finally, evaluate such projects at a scale their evidence can bear. The episode's prediction of a credit collapse within roughly twelve months is a speaker forecast. Nathan's belief that wider use of existing AI could deliver the discussed four-percent growth comes without macroeconomic modelling. A deployment programme can first measure whether people use the service, finish relevant tasks, and continue benefiting. Those observations can justify further trials without turning one successful interaction, or one newly enabled interface, into proof of an economy-wide outcome.

Source

• Original episode: AI:AM: Where's the Steering Wheel? My Agent Set the Password, Hidden Valence, Proofs Find Linux Bugs