🧵Tired of scrolling through your horribly long model traces in VSCode to figure out why your model failed? We made StringSight to fix this: an automated pipeline for analyzing your model outputs at scale. ➡️Demo: stringsight.com ➡️Blog: blog.stringsight.com

The AI community seems to increasingly be heading towards a polarized world when discussing safety and consolidated power. I see this discourse as a false dichotomy, so @profjoeyg and I wrote an essay on how we need to change the conversation (link below).

👀Humans compare images by looking back and forth. Many open-weight VLMs encode each image independently, and defer comparison to the LM. We introduce SVE: Stateful Visual Encoders for Vision-Language Models, where the visual encoder itself becomes change-aware. 🌐Project: statefulvisualencoders.githu… 📰Paper: arxiv.org/abs/2606.04433 💻Code: github.com/StatefulVisualEnc… 1/n

We release Recon — a new approach to reasoning synthesis for user modeling. The key insight: post-hoc rationalization ≠ reasoning. We propose using action reconstruction as a scoring criterion for synthesized reasoning traces, yielding more causally faithful reasoning and improved downstream action prediction across user modeling tasks. Paper and project page in 🧵

🔭 We’re releasing Hodoscope: an open-source tool for unsupervised behavior discovery. It lets you visually explore and compare agent behaviors at scale. It helped us discover a novel reward hacking vulnerability in Commit0 - with just a couple minutes of human effort.

Why does GPT-5.1 Codex score 6.5% worse than GPT-5 Codex on Terminal-Bench, with the same scaffold? 🧵 GPT-5.1 times out at ~2x the rate of GPT-5. Excluding timeouts, GPT-5.1 wins by 7.2%. We analyzed 256M tokens of traces and found this in under an hour. Here’s how 👇

Always great to see more work in model diffing!

I had a fun time writing a deep dive on Diffusion Language Models - with an equation walkthrough and Excalidraw sketches ✏️ In Part 1, I focused on the method: what does “noise” even mean for text, and how do DLMs denoise back into tokens? winterrykim.github.io/blog/2…

We trained diffusion models on a billion LLM activations, and we want you to use them! New preprint: Learning a Generative Meta-Model of LLM Activations Joint work with @feng_jiahai, @trevordarrell, @AlecRad, @JacobSteinhardt. More in thread 🧵

Continual learning from natural language is data-hungry. Can we make it sample-efficient? SIEVE distills natural language context (instructions, feedback, rules, etc.) into model weights using as few as 3 examples only of queries—outperforming prior methods and even in-context learning baselines. (1/n)

LMArena is now Arena. A name that takes us back to our roots with a powerful mission: to measure and advance the frontier of AI for real-world use. We have grown from a small PhD research project to a platform powered by a global community of millions. This rebrand has been shaped by the people who use it. 👇 Take a look inside the rebrand.

With the rise of agents comes the need to better evaluate their true visual capabilities, VisGym takes a step in this direction and analyzes what representations are best for models and how they fail. It was such an honor to be a part of such an incredible team!

✨Thinking with Blender~ Meet VIGA: a multimodal agent that autonomously codes 3D/4D blender scenes from any image, with no human, no training! @berkeley_ai #LLMs #Blender #Agent 🧵1/6

In an effort to better understand VLMs, we found that they are fragile in surprising ways. Just changing the color of pointing markers (red circle → blue circle) can completely change the results! :

This is the kind of work that makes you rethink what leaderboards are actually measuring. If marker color can reorder rankings, are we evaluating vision capability or visual sensitivity to arbitrary details?

Seemingly task-irrelevant details, such as the choice of visual markers, can actually cause large changes in the performance of vision-language models! Check out our work that investigates the fragility of visually prompted benchmarks: lisadunlap.github.io/vpbench…

This is an amazing collaboration between @LongTonyLian, @HavenFeng, Jiahao Shu, @XDWang101, @ren_wang1, @trevordarrell, @alsuhr and @akanazawa!

We release VPBench along with our analysis framework: Datasest: huggingface.co/datasets/long… code: github.com/TonyLianLong/VPBe… Arxiv: arxiv.org/abs/2512.17875

With all these sources of instability, what can we do about it? Well larger dataset sizes help a lot: for instance, the ranking changes you see with JPEG compression decrease a lot when you increase your dataset size. A good rule of thumb is if your confidence intervals are wide: you need more eval data.

Even changes humans can’t perceive matter. JPEG compression quality or tiny numerical differences can change results, while compression for non-visually prompted tasks has little effect.

It gets worse with marker style. Change the marker’s color, shape, size, or text position and accuracy shifts by multiple points. You can literally reshuffle the leaderboard by how you draw the circle.