Jun 17
Post here: alignmentforum.org/posts/GTY… done with @ArthurConmy, @NeelNanda5 the GDM Language Model Interpretability team.
4
245
Interpretability Researcher @ Google DeepMind, ARENA founder. Guinness World Record holder for the largest number of clothes pegs held in one hand (yup, really)
Joined December 2015
- Posts 47
- Following 6
- Followers 1,657
- Likes 10
17 Photos and videos
Jun 17
Caveats: this is one model family at one point in time, and we didn't run full ablations on all findings. Our post shares the pipeline features we had when we eventually got it working, which it didn't have initially. We'd like to see more comprehensive ablations of these.
1
2
264
Jun 17
As one example of capability collapse: since our synthetic chat data had no tool calls, overtraining would make the model worse at tool use, which can sometimes be confused with refusal in evals like ODCV. Models that can't use tools might invent fake reasons why they don't.
1
4
149
Jun 17
Other takeaways: multi-turn adversarial evals surface failures which are harder to see single-turn; midtraining works but is hard to get right (we share what helped in our post); mixing in baseline SFT data prevents capability collapse.
1
2
133
Jun 17
We also found a knowledge/internalisation gap. Models could state target traits accurately (and midtraining boosts this a lot) well before they reliably acted on them. Saying the right thing and doing it are separate, and midtraining can often achieve the former, not the latter.
1
5
144
Jun 17
One lesson to take from this: models can learn artifacts from synthetic data in ways that don't always show up in eval scores, even when you'd expect them to. When possible, it's worth detecting these directly in the data, before you train, rather than only via downstream evals.
1
3
145
Jun 17
We also tried filtering out two patterns (emotional-validation openings, and bottom-line-up-front responses) and retraining. The filtering changed model responses as expected, but didn't move the delusion-validation eval.
1
4
152
Jun 17
...So we built a scan/cluster/autorate pipeline to find over-represented structural patterns in any dataset using an LLM. It's general, and we think it's useful for synthetic-data work broadly, especially model-organism research where training artifacts can confound results.
1
5
178
Jun 17
One interesting failure mode: superficial patterns in synthetic SFT data get reinforced hard. Teaching "ask for clarification when underspecified" taught the model to ask for clarification even on "what's 1 1?". Each example was fine alone; the problem was over-representation.
1
6
186
Jun 17
We evaluate on a suite of safety evals, each deliberately off-distribution from our training data along at least one axis (e.g. multi-turn or agentic). Midtraining usually improves OOD evals on its own; SFT often stacks. Capabilities stay flat.
1
4
319
Jun 17
This work is inspired by Li et al's model spec midtraining (MSM): train on synthetic docs before chat finetuning to shape how the model generalises. We test how well this holds at frontier scale, and share some details we found helpful for getting this training to work.
1
6
4,636
Jun 17
New GDM research from the AGI safety team: can you instill positive traits into a model with synthetic document finetuning? We midtrain Gemini 3 Flash on docs describing the traits we want, then finetune on chat data demonstrating those traits. This pipeline robustly instils the traits, and it generalises OOD 🧵
3
3
60
8,616
Full details in the LessWrong post: lesswrong.com/posts/nQAN2vxv…
Feedback welcome in our Slack: join.slack.com/t/arena-uk/sh…
2
4
38
2,401
We strongly recommend using LLMs to help you work through this material. The value isn't in typing every function - it's in the structure, pedagogical framing, and exercises designed to teach you the meta-level skill of noticing notes of confusion in your own results.
1
21
2,543
We also have a new website at learn.arena.education, with a few features beyond the old Streamlit page:
- Course planner: submit your preferences, get a daily/weekly study breakdown
- Context menu: download exercise content to seed Claude Code projects (or drop it into an LLM)
1
1
29
2,271
Investigator Agents (4.5): starts with replicating AI psychosis results (Tim Hua), then builds a mini-Petri from scratch using inspect-ai to run whistleblowing evals. Ends with using Petri directly, including recent features like eval-awareness from Petri 2.0.
1
1
12
1,905
LLM Psychology & Persona Vectors (4.4): replicate Anthropic's assistant axis work and implement activation capping to prevent persona drift without harming capability. Then build a persona vector extraction pipeline with contrastive prompts and autorater scoring.
1
1
16
1,790
Reasoning Model Interpretability (4.3): replication of the Thought Anchors paper, with blackmail extension. Covers black-box methods (resampling reasoning chunks and measuring downstream effects) and white-box methods (attention analysis causal interventions).
1
1
15
1,785
Science of Misalignment (4.2): two detailed case studies - Palisade's shutdown resistance and Anthropic's alignment faking. Both reinforce the core skill of rigorously testing features of your environment to distinguish genuinely misaligned behavior from more benign explanations.
1
1
13
1,826
Emergent Misalignment (4.1): structured around Soligo & Turner's work training smaller model organisms that exhibit emergent misalignment. Covers autoraters, LoRA finetunes, steering experiments (including diagnosing why they fail), phase transitions, and more.
1
13
1,937