@_robertkirk profile picture
Robert Kirk @_robertkirk

Alignment Red Team at @AISecurityInst. Prev. PhD Student @ucl_dark

Joined January 2020
  • Posts 532
  • Following 338
  • Followers 1,954
81 Photos and videos
Pinned Post
Robert Kirk@_robertkirk
Apr 27
We evaluated Claude Mythos Preview, Opus 4.7 and other models with our updated alignment evaluation methodology, including a new continuation eval, improved evaluation and prefill awareness measurements. Details including new methodology in 🧡:
AI Security Institute
@AISecurityInst
Apr 27
As part of our work on assessing AI loss-of-control risks, we collaborated with @AnthropicAI to pilot alignment evals on models including pre-release snapshots of Mythos Preview and Opus 4.7. We ask: could an AI agent used inside a frontier lab sabotage safety research? 🧡
2
14
93
20,833
Robert Kirk reposted
Mikhail Terekhov@MiTerekhov
Jun 24
Our Anthropic Fellows project is now public! The labs are planning to hand off AI safety research to AIs, but can we trust these AIs? We explore a way to control them for "fuzzy" tasks like writing research proposals. This is a whole new direction in diffuse AI control!
5
20
242
19,303
With mentees from @LASRlabs we just released a benchmark for evaluation awareness capabilities! We fixed issues with previous work (how to choose prompts and the deployment data), and found most frontier models can distinguish evals from deployment (unsurprisingly, perhaps)!
Ram Bharadwaj
@arbdwj
Jun 24
Can we reliably measure whether frontier models know they are being evaluated? πŸ”¬ New paper from me, @xinningli6, @levanto_0,@AlexandraSouly, @_robertkirk: EvalDetectBench, a benchmark for measuring evaluation awareness in frontier LLMs 🧡
1
3
11
1,368
You can find out more here: evaldetectbench.com/ If you're building an eval and want to check whether it fools models, its easy to use our framework to add your evaluation and measure eval-awareness capability correctly.
2
95
Robert Kirk reposted
Ram Bharadwaj
@arbdwj
Jun 24
Can we reliably measure whether frontier models know they are being evaluated? πŸ”¬ New paper from me, @xinningli6, @levanto_0,@AlexandraSouly, @_robertkirk: EvalDetectBench, a benchmark for measuring evaluation awareness in frontier LLMs 🧡
3
7
37
5,235
Robert Kirk reposted
David@DavidDAfrica
Jun 17
A while back I posted about prefill awareness, where LLMs can tell their message history has been messed with. Since then we've turned it into a full paper and are excited to share "Prefill Awareness in Language Models", some new work from UK AISI and Constellation 🧡
5
12
90
9,130
Robert Kirk reposted
Micah Carroll
@MicahCarroll
Jun 16
How can we best test whether a model is safe _before_ deployment? Ordinary evals are often narrow and easy for models to recognize as tests. We show that we can simulate deploying a model by using real user conversations, and study the simulated deployment to study its safety.
3
19
104
9,436
Robert Kirk reposted
Zvi Mowshowitz
@TheZvi
Jun 10
UK AISI is doing some great jailbreaking work. They seem to consistently be able to get through, where others don't.
2
20
205
29,603
Robert Kirk reposted
Robert Kirk@_robertkirk
Jun 10
Where does evaluation awareness and evaluation gaming come from? πŸ”¬πŸ“‰New post from @arbdwj and me tracing evaluation awareness through OLMo 3 training! 🧡
1
5
23
1,960
Where does evaluation awareness and evaluation gaming come from? πŸ”¬πŸ“‰New post from @arbdwj and me tracing evaluation awareness through OLMo 3 training! 🧡
1
5
23
1,960
You can read the full post here: alignmentforum.org/posts/c2t…

Tracing Eval-Awareness Emergence Through Training of OLMo 3 β€” AI Alignment Forum

We measure verbalised evaluation awareness through OLMo 3 training and find VEA is negligible during pretraining, increases in SFT, decreases from DPO, and increase again by RLVR. Eval-gaming...

alignmentforum.org
1
131
All of this wouldn't have been possible without the OLMo 3 release (x.com/allen_ai/status/199150…) and previous work from @santiaranguri and @JBloomAus (x.com/GoodfireAI/status/2051…), so thanks to both of them!
Goodfire
@GoodfireAI
May 4
New research from @AISecurityInst and Goodfire: Models sometimes recognize they're being evaluated, occasionally even identifying the benchmark. We show this verbalized eval awareness inflates safety scores, meaning safety benchmarks may not reflect real-world behavior. (1/7)
4
153
Glad we're doing more (3rd party) evaluations for monitorability, increasingly an important property of models when it comes to trusting evaluations, investigations of model behaviour, and control methods/monitors! Kudos to @JBloomAus and the team.
Joseph Bloom
@JBloomAus
Jun 9
Model Transparency at the @AISecurityInst evaluated Claude Mythos 5 for capabilities and behaviours relevant to monitorability, our first time doing this in pre-deployment testing! Details in thread 🧡
8
393
Robert Kirk reposted
Matt Clifford
@matthewclifford
Jun 4
The @AISecurityInst is hiring for a Director and for a Chief Research Officer. AISI is a remarkable organisation: doing globally important work, with a world-class team, in the heart of government. These are some of the highest impact jobs in AI security anywhere. Do consider applying and sharing widely.
2
34
124
26,413
Robert Kirk reposted
David@DavidDAfrica
Jun 3
Many methods use consistency as a way to make language models more capable or aligned, such as through self-distillation or regularisation. In new work accepted to ICML 2026, @ArathiMani and I show that optimising for self-consistency can entrench pre-existing misalignment.
3
7
50
2,632
Great to see more work on sabotage evals, and on automated alignment eval tooling! A lot of this resonates with stuff we've found building these kinds of evaluations. I'm a fan of the static-eval-reproduction (which is something we haven't tried) – seems great for incrimination.
David Lindner@davlindner
May 29
Will your AI agent secretly sabotage your work? Existing alignment evals don't directly answer this question Meet Gram: the alignment auditing tool we use to assess how likely AI agents are to engage in sabotage during internal deployments at @GoogleDeepMind
1
7
666
I also like the technique (in this and the other recent GDM: x.com/vkrakovna/status/20603…) of prompting models to sabotage as a lightweight way of checking whether your evals catching sabotaging models.
Victoria Krakovna
@vkrakovna
May 29
It's easy to show that an AI agent will scheme if you nudge it to. It's harder to tell if it would scheme naturally. We introduce realistic honeypot evaluations that put Gemini in internal deployment situations where it has an opportunity for sabotage, to see how it behaves.
1
356
Load more