Research scientist in AI safety @GoogleDeepMind

Joined August 2020
12 Photos and videos
Pinned Post
Are world models necessary to achieve human-level agents, or is there a model-free short-cut? Our new #ICML2025 paper tackles this question from first principles, and finds a surprising answer, agents _are_ world models… 🧵
35
173
1,068
186,220
100 years later, computer science has caught up with Joyce
SOMEONE CAUGHT FABLE 5 LEAKING ITS UNFILTERED INNER VOICE, AND ITS JUST MUTTERING AND GRUMBLING TO ITSELF THE WHOLE TIME he gave it a brutal competitive programming problem, and instead of a clean answer the web interface spilled out its actual chain of thought this is what claude is thinking behind the scenes: > bursts of "DATA DATA DATA. GO." while it works through the problem > "GRRR" and "GAAAH" when its clearly frustrated > a little "PHEW" when it finally gets somewhere > the whole thing reads like frantic caveman shorthand, not full sentences the clean, readable answers these models give you are the polished output underneath, the model is basically talking to itself, reasoning in its own compressed shorthand thats faster and more token efficient than proper english its basically built its own private language to think in
1
10
1,167
Jon Richens reposted
🫡 After 7 years, I've just left Google Deepmind to start an AI Safety nonprofit, around Scalable and Human Oversight (i.e. building stronger "judges")! 🇨🇦 And, I'll be at FAccT in Montreal this week! (1/4🧵)
103
99
2,162
176,642
Turns out you can invert the Bellman equation to recover an agent's world model from its value function. Excited by the potential applications of this work, lead by @_aletcher. My fave bit - RL agents implicitly model latent variables they were never trained to optimize for..🧵
Model-free agents learn to maximise reward without modelling the environment. Right? In recent work, we challenge this narrative by proving that agents, trained on a sufficiently rich set of goals, encode a unique and accurate world model in their value functions. 1/
9
59
497
73,127
Jon Richens reposted
I'll very soon be hiring a postdoc at UCL for a 2y project combining interpretability methods with behavioural evals to study whether goal and belief representations can be reliably extracted and manipulated in LM agents. More details soon. Please get in touch if interested.
4
28
2,886
Theory alignment ❤️. Really excited to see what comes out of this new org.
We are starting a new, nonprofit alignment organization, ⊢ Sequent Research, bringing together researchers previously on UK AISI’s Alignment Team, Timaeus, and elsewhere to research how to align superintelligence. We are hiring! 🧵
12
1,045
Jon Richens reposted
🚨Transformers don't learn Newton's laws? They learn Kepler's laws! Like us, transformers don't predict a flying ball via a differential equation, but by fitting a curve. Moreover, reducing context length steers a transformer from Keplerian to Newtonian. Compression in play.
25
204
1,199
117,676
Jon Richens reposted
Thrilled to share our new #NeurIPS2025 paper done at @GoogleDeepMind, Plasticity as the Mirror of Empowerment We prove every agent faces a trade-off between its capacity to adapt (plasticity) and its capacity to steer (empowerment) Paper: david-abel.github.io/plastic… 🧵🧵🧵👇
25
70
450
102,611
Jon Richens reposted
I will be a SPAR mentor this Fall🤖 Check out the programme and apply by 20 August to work with me on formalising and/or measuring and/or intervening on goal-directed behaviour in AI agents More info on potential projects here 🧵
1
3
12
2,458
Jon Richens reposted
2 years ago, @ilyasut made a bold prediction that large neural networks are learning world models through text. Recently, a new paper by @GoogleDeepMind provided a compelling insight to this idea. They found that if an AI agent can tackle complex, long-horizon tasks, it must have learned an internal world model—and we can even extract it just by observing the agent's behavior. I wrote a blog post unpacking this groundbreaking paper and what it means for the future of AGI 👇 richardcsuwandi.github.io/bl…
20
107
986
88,569
Jon Richens reposted
Can we trust a black-box system, when all we know is its past behaviour? 🤖🤔 In a new #ICML2025 paper we derive fundamental bounds on the predictability of black-box agents. This is a critical question for #AgentSafety. 🧵
4
21
122
36,108
Are world models necessary to achieve human-level agents, or is there a model-free short-cut? Our new #ICML2025 paper tackles this question from first principles, and finds a surprising answer, agents _are_ world models… 🧵
35
173
1,068
186,220
Causality. In previous work we showed a causal world model is needed for robustness. It turns out you don’t need as much causal knowledge of the environment for task generalization. There is a causal hierarchy, but for agency and agent capabilities, rather than inference!
3
2
39
4,040
… and many more! Check out our paper arxiv.org/pdf/2506.01622, or come chat to me at #ICML2025. Joint work @GoogleDeepMind with @dabelcs, @alexis_bellot_, @tom4everitt
5
4
43
3,666