@bleysg profile picture
Bleys Goodson @bleysg

Helping people engineer the future.

Joined March 2009
  • Posts 217
  • Following 1,312
  • Followers 732
33 Photos and videos
Pinned Post
Bleys Goodson
 @bleysg
May 26
I have been investigating LLM serving economics and here are my METR-anchored projections of model capabilities as Vera Rubin ( comparable TPUs) and next-gen Cerebras are deployed over the next couple of years. Relevant background: 10 years managing large, global R&D datacenter efficiency and the last 3 years working every angle of LLM engineering. (v2 post)
4
18
1,829
GPT-5.6 Sol might drop tomorrow. How big is it? They've announced it will later have a fast tier powered by Cerebras generating up to 750 tokens/second. I know enough about Cerebras and large model architectures to reverse engineer the size. How big do you think it is? 🧵👇
2% Under 1T (like GLM-5.2)
9% 1T - 2T (like DeepSeek4)
70% 2T - 4T
19% Over 4T
0%
0%
0%
0%
47 votes • 1 day
1
10
887
We know the likely maximum cluster size is 88 wafers, because that's the max shown in the Cerebras datasheet:
1
7
402
I crunched the numbers for inference costs with my own TCO calculator. Optimistic throughput here shows they'd make a solid 17x or more TCO if they charged 3x the slow rate for Sol. Worst case estimates for throughput paired with only a 50% premium over slow Sol still puts them at around a 4x 5 year TCO.
7
314
With the fastest spec of the M5 Max now being $7000, it's the same price as 2x DGX Spark. The Mac has the advantage still if you need something to travel with (AI on flights being a top case). Otherwise, now 2x Sparks are looking better than ever. DeepSeek V4 Flash runs at effectively lossless quality on 2x DGX Spark at 40 tok/s (90 with multiple agents) compared to 25 tok/s on M5 Max, with less ability to run agents or background work like document RAG. Also, most tokens processed in everyday workflows are input/prefill. When loading documents, doing web search, etc. On the Sparks, that happens at over 1200 tok/s, on M5 Max it would be 300. 4X slower for 80 % of tokens! You can run bigger models faster, and not have fans screaming in your face and a hot lap.
3
9
4,796
On Mac, DeepSeek 4 Flash only runs as a 2-bit quant too, which is still effective, but meaningfully degraded quality vs the true model you can run on the Sparks.
1
4
384
NVIDIA has committed to refreshing the Spark platform with every generation, and publicly stated a Vera Rubin Spark is coming in 2028. 🤞
4
1
32
27,274
A few new names in here I'm happy to follow. Autoresearch is a burgeoning subfield which will be a defining feature of the next couple of years.
Lillian Ma
 @lillian_ma_
Jun 22
Emerging autoresearch labs worth following: @AutoScienceAI (@eliot_cowan) One of the cleanest “AI builds AI” bets: agents that invent, test, and ship ML models; not just tune hyperparameters. @intology (@zhouandy_) Zochi and Locus are built for the full research loop: read, hypothesize, code, run thousands of experiments, learn from failures, repeat. @thesis_labs (@eigentopology) YC F25. Treating ML research as a compounding search problem, where every experiment improves the next one instead of dying in a Notion doc. @Recursive_SI (@RichardSocher, @_rockt, @jeffclune) A very ambitious new bet on AI systems that run open-ended experiments on how to make AI systems better. autoresearch with recursive consequences. @EdisonSci (@SGRodriques, @andrewwhite01) A newer FutureHouse spinout bringing AI scientists into biopharma R&D — where “deep research” has to survive real data, real experiments, and real timelines. @HarmonicMath (@tachim, @vladtenev) Math’s version of autoresearch: AI exploring new proofs, with formal verification as the anti-hallucination layer. @readysetpotato (@Nick___Edwards) An AI scientist for actual research workflows — papers, hypotheses, protocols, computational tools, and eventually lab automation. @EvoScientist (@_xizhang) A very early one to watch: multi-agent AI scientists with persistent memory, so failed ideas and experiments improve the next research cycle. @SakanaAILabs (@hardmaru) The original AI Scientist builders. Still one of the best technical follows for research artifacts, open source, and weird ideas that actually run. not exhaustive - add the early teams I’m missing in the comment
2
455
I'm in the middle of a research campaign which I can now say fairly strongly is a major upset to the pareto frontier for general purpose LLM quantization on all fronts: speed, size, and quality. The wild part I'm just discovering in the last couple days of labs is it surpasses QAT versions of the same models. The QAT I'm doing is categorically an improvement vs Gemma 4 QAT releases. Not only that, but it's also -- on high-end consumer hardware -- an improvement over DiffusionGemma in single-stream speed! More to come soon.
3
289
Gemma 4 12B new QAT release allows you to run high context comfortably on 16GB GPUs, while sacrificing very little in quality vs the unquantized model. llama.cpp just got support for MTP w/ Gemma 4 too. I tested performance. RTX 4090 gets a boost from 90 tok/s to 120 tok/s. PRO 6000 (should be similar for RTX 5090) goes from 120 to 155 tok/s. Prefill takes a small hit, but stays in the 5000 range. This model is notable for being one of the most capable multimodal models available today which will run on a single mid-tier GPU faster than you'd get on API for most models. Most model speed testing in the consumer space isn't as thorough and visual as the sweeps I've put together here. I built my own pipeline for this testing. If you'd like to see more or have me share it, let me know. @googlegemma
4
9
1,725
Thanks to @UnslothAI for the quants: unsloth.ai/docs/models/gemma…

Gemma 4 QAT | Unsloth Documentation

Run Google Gemma 4 QAT models locally, including E2B, E4B, 12B, 26B-A4B, and 31B.

unsloth.ai
185
Gemma4 MTP in llama.cpp details: github.com/ggml-org/llama.cp…

llama : add Gemma4 MTP by am17an · Pull Request #23398 · ggml-org/llama.cpp

Overview This PR adds MTP support for Gemma 4 models. For the MoE model I don't observe a speed-up on my system, but the dense model has on average >2x speedup. Correctness wise I a...

github.com
1
191
Full model details: blog.google/innovation-and-a…

Introducing Gemma 4 12B: a unified, encoder-free multimodal model

An overview of Gemma 4 12B, a model designed to bring high-performance multimodal intelligence directly to your laptop.

blog.google
140
QAT takes the model weights down from 27 GB to 7GB (in use, with context full, you'll still need ~13 GB VRAM)
1
2
268
12B model is comparable in performance to the 26B-A4B MoE model across many tasks. 12B model benefits from MTP, while the MoE does not though.
2
204
New Agent @arena pits models against each other in a real agentic harness. Congrats, dynamic real-world comparison is here! I spotted a big next opportunity.. Their analysis of the tasks and categories is much like my analysis of DeepSWE challenge catalog: x.com/bleysg/status/20622801… The big opportunity: Use task distribution and the deeper categories I broke out in the DeepSWE analysis to build a by-model signal on things like.. "Highest-complexity tasks which are full feature requests in Rust with a behavior theme of data transformation" This then goes from, "average performance across all work types, in all languages, for all users," to, "average performance for my work types, in my languages, for my users" ... the signal people actually want.
Bleys Goodson
 @bleysg
Jun 3
I got a lot of followup on my DeepSWE testing of Minimax M3 asking what it means to be fluent in this eval set. I dug into it. Full report covers breakdown by languages, task types, complexity, and more so you can see just how applicable it is to your type of work. entrpi.github.io/misc/deepsw…
1
7
491
Agent Arena details: x.com/arena/status/206256674… or arena.ai/blog/agent-arena-me…
Arena.ai
 @arena
Jun 4
Introducing Agent Arena: real-world agentic evals at scale. How do you evaluate agents doing actual work? We measure millions of live sessions where real users accomplish real tasks. On Arena, models now get web search, filesystem, and terminal tools to complete complex workflows: writing code, creating slide deck, researching the web, building apps, and analyzing documents. Every session produces rich signals. Users iterate with the agent turn-by-turn: approving, editing, correcting, praise or expressing frustration. The environment gives feedback too: shell errors, tool failures, recovery attempts, and more. Our leaderboard measures each model's agentic performance using causal inference across five signals: task success, steerability, error recovery, user praise vs. complaint, and tool hallucination. This leaderboard snapshot is built from 300K tasks, 2M tool calls, and 40M lines of code by agents. Top labs in Agent Arena: - #1 @OpenAI: GPT-5.5 (High) - #2 @AnthropicAI: Claude-Opus-4.7 (Thinking) - #3 @Zai_org: GLM-5.1 - #4 @GoogleDeepMind: Gemini-3.1-Pro - #5 @Kimi_Moonshot: Kimi-K2.6 More analysis in the thread, with the full technical blog below.
1
225
I got a lot of followup on my DeepSWE testing of Minimax M3 asking what it means to be fluent in this eval set. I dug into it. Full report covers breakdown by languages, task types, complexity, and more so you can see just how applicable it is to your type of work. entrpi.github.io/misc/deepsw…
4
2
81
7,768
Every challenge is searchable with readable summaries at the end of the report as well.
5
1,007
There is a backstory to why @NVIDIAAI has stuck to 10% throughout the Nemotron 3 series, including the new 550B Ultra model, while most of the industry chases MoE with 3-5% activation. LatentMoE is that story. They argue effective MoEs be evaluated by two dimensions: accuracy per FLOP and accuracy per parameter. The race toward 3-5% activation implicitly optimizes only the first. arxiv.org/abs/2601.18089v1
1
1
14
2,257
When you are flop-rich with GB200, Nemotron-style architectures better optimize for the hardware you have available. This may make Nemotron 3 Ultra the best blend of intelligence to tok/s to tok/megawatt of any model available this month.
3
663
Load more