Building with local AI & LLMs | Insights, tools & honest experiments

Joined July 2022
142 Photos and videos
Pinned Post
What’s the best model you can run on your @NVIDIAAI DGX Spark? 1× DGX Spark * ⁠Qwen 3.6 35b NVFP4 - 256k ctx, 110 tok/s * DeepSeek v4 Flash REAP * ⁠Qwen 3.6 27b - 256k ctx 19, tok/s 2× DGX Sparks ← sweet spot! * DeepSeek v4 Flash - 1M ctx, 40-45 tok/s * Step-3.7-Flash - 256k ctx, image support, 30 tok/s 4× DGX Sparks * GLM 5.2 NVFP4 on all 4 * 2× DeepSeek v4 Flash setups, each on 2× DGX Sparks. Links and repos below 👇
74
64
717
65,471
WOW, an important release from NVIDIA 🔥 NVIDIA Model Optimizer 0.45.0 is out — big upgrades for everyone, but especially exciting for MoE models with NVFP4 quantization! Key benefits for MoE quantization: ✨ New “active experts” mode in AutoQuantize — now understands only some experts run at a time, for smarter mixed-precision choices ✨ Better quantization for DeepSeek-style models easy exact MXFP4 → NVFP4 conversion for routed experts ✨ New W4A16 NVFP4 weight-only mode (no calibration needed) — huge memory savings while keeping good quality ✨ Improved calibration & KV-cache options tailored for MoE models Should result in higher-quality NVFP4 MoE models, lower memory use, and better throughput on Blackwell with almost no accuracy loss! This might fix the issues I'm having with Qwen3.6-35b NVFP4 if they are generous enough to update it. Full notes: github.com/NVIDIA/Model-Opti…
15
414
Cant wait to test Hy3 seems very promising.
Hy3 from @TencentHunyuan is out. Great to see real-world workflows emphasized, with GLM-5.1 cited in the comparison: "we ran a blind test with 270 experts from various disciplines, working on real-world workflows. Hy3 scored 2.67/4, outperforming GLM-5.1 at 2.51/4."
3
21
1,846
Which local model is the best for Agentic Workflows for a single @NVIDIAAI DGX Spark? (or any other 96-128gb VRAM rig) After running 84 scenarios, 16 categories, 8 trials each, on Hermes-Agent style multi-turn tool orchestration, there is a very clear winner. 🏆 Qwen 3.6 35B A3B Q8_K_XL is #1 It’s the only model that hit perfect scores across the board with zero catastrophic failures. The Full Ranking: Qwen 3.6 35B A3B UD Q8_K_XL — 91.0 Qwen 3.6 27B NVFP4 — 89.0 Qwopus 3.6 27B Coder MTP — 85.2 DeepSeek V4 Flash Q2 — 86.5 Agents-A1 Q8_0 — 83.4 Gemma 4 26B — 81.4 Nemotron 3 Nano Omni 30B — 79.0 Bottom line: If you’re running agents locally on a DGX Spark or any 96-128GB rig in 2026, Qwen 3.6 35B Q8_K_XL is currently the move. Full report deep dive 👇 github.com/MiaAI-Lab/Best-Lo…
32
12
190
7,074
I'm gonna post a simple recipe for Qwen 3.6 35B A3B Q8_K_XL soon
1
13
505
For anyone asking why not the 35b nvfp4... x.com/MiaAI_lab/status/20738…
Contrary to @NVIDIAAI's excellent Qwen3.6-27B NVFP4, the Qwen3.6-35B NVFP4 is performing poorly compared to @UnslothAI’s Qwen3.6-35B GGUF in agentic workflows. I also tried @RedHat_AI’s NVFP4 version of the 35B and it’s bad as well. Something seems broken with the NVFP4 quantizations for the 35B model. ☹️
2
6
761
Wanna save precious tokens when using /goal ? ✨Add this to your goal prompt✨ - do not send progress updates while working. - only message me when the task is complete, blocked, or requires a password/manual action. - keep the final report short: status, final health result, final chat response, and any files changed. - use minimal tokens for anything that is not evidence or an action taken. Works well on Codex/Claude or any other model with /goal
3
1
51
1,951
Mia reposted
Wouldn't it be great... @Alibaba_Qwen wake up 😴
40
25
576
77,313
Mia reposted
The older I get, the more I realize that one of the most attractive traits is genuine enthusiasm. It's energizing to spend time around people who show real excitement for life. For people, ideas, and tiny moments. It takes courage to care so openly. Enthusiasm is contagious.
233
979
7,570
221,619
Nemotron-3-Nano-Omni-30B-A3B smashing 80 concurrent sessions on a single @NVIDIAAI DGX Spark! 🔥 256k context. MTP enabled. All sessions running completely different prompts. ~800 tokens/s cumulative 🤯 Video starts slow but goes hard — watch till the end!
35
29
294
18,992
Contrary to @NVIDIAAI's excellent Qwen3.6-27B NVFP4, the Qwen3.6-35B NVFP4 is performing poorly compared to @UnslothAI’s Qwen3.6-35B GGUF in agentic workflows. I also tried @RedHat_AI’s NVFP4 version of the 35B and it’s bad as well. Something seems broken with the NVFP4 quantizations for the 35B model. ☹️
40
6
173
13,123
Run Gemma-4-26B-A4B on your DGX Spark 256k context • MTP • up to 18 concurrent sessions 🚀 Performance: 1 session → 30 tok/s 2 sessions → 63 tok/s 4 sessions → 109 tok/s 8 sessions → 182 tok/s 18 sessions → ~272 tok/s Get the full recipe here: github.com/MiaAI-Lab/Gemma-4…
20
13
154
7,632
kv cache numbers
1
9
878
Btw, if you reduce context to 128k you could do 36 concurrent sessions 🤯
3
711
Watch Gemma-4-26B running 18 concurrent sessions on a single @NVIDIAAI DGX Spark — delivering a cumulative 300 tokens per second. 🔥
48
26
367
21,558
It this blows your mind wait for the next video 👀
18
1,231
ttft was almost instant on all 18 sessions
1
13
1,473
This is mind blowing 🤯 GLM 5.2 on 4× @NVIDIAAI DGX Sparks at 30 tokens per second. I’m not aware of any better local setup for running GLM 5.2 this cheaply!
🧠⚡ Running GLM-5.2 at home the FULL 744B, all 256 experts, UNPRUNED across 4× NVIDIA DGX Spark (GB10). 200K context · MTP spec-decode · fp8 sparse-MLA KV · vLLM native multi-node ~30 tok/s single-stream, 60 tok/s @ 6 concurrent. Full replicable recipe, open-sourced 👇 github.com/tonyd2wild/GLM-5.…
19
16
353
38,998
I'm on a quest to find the best local model to run on a single NVIDIA DGX Spark, specifically for agentic workflows in @NousResearch's Hermes Agent. Which model do you think would handle it best? Coming soon ⏳
67
7
240
17,143
Mia reposted
Hermes Agent is built for sovereignty and constructing your AI stack how you want and need it to be. No vendor lockins, no model limitations, and most importantly, your IP is built through the self improvement loop, automatically. Hermes sets you free 🪽
Our thoughts on the importance of AI sovereignty. 1. Your AI sovereignty dictates your institution’s future. Sovereignty is the precondition for choice. Relinquishing sovereignty transfers the future choices of your institution to others, who are likely to exploit it for their gain and your loss. 2. Data retention is your treasure. Transfer it at your own peril. Your ability to win is dictated by your ability to recognize and use your unique edges, and you keep winning by compounding the underlying data to generate new insights. Transferring that data hands over access to your pre-existing winning plays and yields the means of production for new ones. 3. Tokenmaxxing hijacks your value orientation and decreases your institutional fortitude and intelligence. The pursuit of high token usage incentivizes disposable scripts over robust software — with the addictive feeling of false progress. There is a reason why those selling tokens refuse to charge based on value. 4. Controlling your weights is controlling your fate. Weights are the distilled form of hard-won, accumulated institutional knowledge. If you let others control your weights, you are allowing them to migrate the alpha of your business to theirs. 5. There is no contradiction between sovereignty and alpha. The architecture that maximally preserves sovereignty is one that enables institutions to own their tribal knowledge, and to compound it as alpha. 6. Politicizing the technical issues involving sovereignty is what your adversary wants. Techno-politicization is the wellspring of false sovereignty. Techno-politicization drives decisions that seem to reduce dependency, but ultimately limit agency — especially on the battlefield in the West. 7. Real expertise is existential. Allowing politics or favoritism to determine your technical decisions rewards whoever is best at politics, not whoever is right. Listen to those closest to the problems, not those speaking most compellingly about them. 8. Learn from institutions that are winning or that have consistently delivered. Institutions facing existential threats do not have the luxury of making technical decisions based on political preferences. 9. Only listen to institutions, countries, and people who have a proven record of being right. A track record of correctness is the best and only signal for future correctness. Judging something as right or wrong based on who you like is exceedingly misguided.
78
138
1,775
143,819
This is so freaking cool!!!
just wired the robodog as probably the first physical @NousResearch hermes /pet I’ve seen around 🐶🤖 approvals are definitely my favorite animation 😂 also it spontaneously offered to speak out loud through the dog next step: object detection for hermes, stay tuned 👀
4
41
4,061