Really excited about this work, led by the wonderful
@CarlGuo866!
When training models to explain themselves, the training itself shifts model behaviors so the original explanation labels become stale. Surprisingly, the model actually *learns to explain its current self* and not the stale checkpoint that generated its training labels. This inclination to self-explain is so strong that *even when trained on another model's explanation labels, the model still learns to explain itself* better than the label source!
I’m excited about this on two fronts: first, it’s indication that models genuinely learn something “introspection-esque”, exploiting privileged access to themselves when generating explanations. Second, practically speaking, it shows self-explanations can directly be inserted into post-training without worrying about expensive label refresh.
Check out
@CarlGuo866’s thread for more details! We pinpoint when this “introspective coupling” happens, and show that this training yields faithful self-explanations even in the presence of auxiliary fine-tuning. 👇
New Paper 📄: LMs just want to explain themselves!
When we SFT an LM on explanations of its own behaviors, do they learn to actually introspect, or do they merely imitate the original training distribution?
We find evidence for the former.
Despite training on a static set of explanations from a base model, the SFT-ed model explains its own current behaviors better than the base model’s behaviors, tracking behavioral drift even when we don’t explicitly train it to.
We call this introspective coupling: self-explanations track a model’s own behavior as that behavior changes, and it shows promise in making introspection training a part of scalable post-training pipelines. 🧵
