Such an iconic scene in Breaking IPO

Now run evals on it and prove it's actually worth it for the intelligence performance hit you're going to have by doing this.

Dang, if only summer would just tell us before it shows up, we could better prepare the electrical grids. I'm sure Mark's got his AC set to 80*, doing his part!

If you want to see the per-task performance in detail per model, and read the full report: hkinsley.com/reflections/in-… I'd like to start digging more into the full responses to this benchmark too, but I think that will deserve it's own report entirely. I am also tempted to inspect some of the providers on openrouter to see who is strongest on actual intelligence per dollar or something. @pingToven I think you should gimme some credits for this. ok thx for reading bye!

There are obviously MANY variables here and only 1 specific benchmark. Running local is hard because of the # of hardware and software variables that come into play. For example all my speeds with vLLM TP would be doubled if I was running on PCIe 5.0 except maybe the concurrency parallelism. My board is 4.0, but the usage of risers to make the cards fit at all has me dropping down to PCIe 3.0 due to signal degradation. For pipeline parallelism, this has very little impact, but it's brutal on the TP. I would also see a jump in perf if I was using a higher memory fraction. I need space on my display GPU for now. Eventually, i'd like to build a dedicated AI server, then I can run these closer to something like 0.95 gpu mem. I also am not certain what is causing the 5 task delta between my DSV4F T2.1 score and official scores. I've seen it validated by 3rd parties to be 61.8% vs my 56% finding. Potentially it's the DSV4F vLLM implementation, maybe MTP=2, or maybe my harness? who knows. This is why you always need to actually test based on YOUR hardware and YOUR software to know for sure. This also makes me highly suspicious of any cloud API provider. Many share their precision, but you have NO clue whats the KV cache? What's MPT/speculative decoding? Every variable matters and I suspect there's likely a decent amount of variance in bench perf from each provider on openrouter. Someone should probably look into that :) it's too easy to see a model, see tok/sec, and go with that provider. Waaaaaaay more questions you should be asking!

In the end, DSV4F outperforms GLM 5.2 IQ4 in intelligence and is much faster, at least on Terminal Bench v2.1. Will be daily driving this for a bit to see how I feel. GLM 5.2 is just such a cool model, but I am now quantifying just how much is lost from native precision.

DeepSeek wants to use 2.5x the tokens of GLM 5.2 IQ4, but it's 5x faster bs=1. The moment we introduce concurrency, vLLMs concurrency takes off. I consistently see a dip for bs=2 on vLLM TP, but it picks up again after. Anyone care to explain this to me? MPT=2 or smth else?

A big factor here though is how many tokens does it take for a solution? If you have 2x tok/sec, but you need to use 3x more tokens to reach a solution, it's not better. Time is the real question.

Detailed look in model performances on Terminal Bench v2.1. GLM 5.2 takes a big hit from native to FP8. Another hit from FP8 to IQ4_NL. Obviously a hit from 4 to 2 bit, but this trade-off might be worth it. Makes me want to test the 1bit. Cool seeing how capabilities degrade.

Then I had a curious question about KV cache. Many people assume 8-bit KV cache is free context locally. But the moment I started playing with GLM 5.2 4-bit w/ 8-bit KV, I felt something was very wrong. Through digging, I eventually found that 8-bit KV was slamming perf. You drop from 46/89 down to 21/89 just for going to 8 bit KV. You're better off going with 2 bit GLM 5.2!

The next question was: GLM 5.2 2-bit (254GB) vs MiniMax M3 4-bit (265GB) Despite being a bit smaller, GLM 5.2 2-bit is actually considerably better than 4-bit MiniMax M3. This was a really interesting result. I had suspected this would be the case, but neat to confirm it.

what matters is intelligence and usefulness. The end goal isn't actually tokens/sec or fitting some particular model into memory w some quant, we want the intelligence. First question I had was 2 bit GLM 5.2 vs 4 bit GLM 5.2. For the footprint delta, this was impressive to me:

Full report: hkinsley.com/reflections/in-… The first consideration is how big of a model can you fit. In my case, my max GPU mem is 384GB. We need some space for context too. These are just a few of my options at various precision and inference speeds.

I set out to figure out which GLM 5.2 quant to run local based on speed and intelligence. Naturally, I ended up selecting DeepSeek V4 Flash and learning a bunch on the way. tldr: Terminal Bench v2.1 scores from local inference (other than FP8 GLM 5.2 baseline from openrouter)

The only ppl using this model will be people who are thinking purely in terms of cost per token instead of cost per useful output. What a weird release.

Just make sure you're only doing the science ant wants you to do!

Love this!

the best thing you can do right now is to stop paying money to the company actively lobbying against your freedom.

I think many are in denial about the restriction of open source ai and banning chinese matmuls. they'll just start throwing people in prison. average public is scared of ai and also doesn't like it. they blame ai for inflation/job loss. A jury trial wont likely go well.

For more information, and to submit a proposal: luckyrobots.com/fellowship If you're looking for a more entry style fellowship, that exists as well, same URL above. As always my DMs are also open too!