Subliminal learning is when LLMs transmit traits (e.g. loving cats) through seemingly meaningless data. What’s going on? We find a simple explanation: it's just steering vector distillation. We explain which traits transfer and why subliminal learning fails across models.

How does subliminal learning - where fine tuning on meaningless data transfers traits between models - happen? Some great insights into how this is closely related to learning steering vectors for the same traits, by @camila_blank and @chewing_a_gum

Updated paper link: arxiv.org/abs/2606.00995

In our new paper, we find an explanation of why subliminal learning occurs. As ever, steering vectors!

I had a lot of fun working on this paper - we found an elegant story for why subliminal learning happens! A key intuition in interpretability is that basically every interesting phenomena in LLMs boils down to adding a steering vector. Subliminal learning is no exception!

This work was done jointly with @chewing_a_gum and advised by @sen_r, @ArthurConmy, and @NeelNanda5, with support from @MATSprogram. Thanks to @cloud_kx and Matt Clarke for helpful and interesting conversations!

Check out our full paper on arXiv for more results and details: arxiv.org/abs/2606.00995v1 Code: github.com/agu18dec/steering…

We show the above holds in settings beyond number sequences, including code and paraphrasing datasets, and with a fine-tuned teacher.

Surprisingly, subliminal learning only works when using adaptive optimizers like Adam, which prevent outlier gradients from dominating this signal. Also, we find subliminal learning only occurs under low-rank training and not under full fine-tuning.

Mechanistically, steering vector distillation happens because gradients on steered data subtly point along the steering direction.

Our explanation helps us make nontrivial predictions, like which traits can be subliminally learned (e.g. for the ~half of animals where steering at inference time doesn’t induce the trait, there’s no subliminal learning).

But how exactly does this explain SL? Steering vectors have both semantic and non-semantic effects. The non-semantic effects in the number sequences make the student learn the vector, including its semantic effects. This explains why SL doesn’t transfer between models.

*Steering vector distillation* is when a student trained on the outputs of a steered teacher learns to imitate that steering vector. This is the mechanism that creates subliminal learning, and it’s also a general phenomenon: even random vectors can be distilled into the student

Steering vectors are directions that, when added to the residual stream of a model, shift its behavior. They can cause strange effects, including subliminal learning: we show there's a vector that's both necessary and sufficient for inducing subliminal learning.