Maybe I'll try these days Atlas Inference as it shows 131 t/s single stream for the 35B with MTP and the guy said that that data is outdated so it will hopefully be even faster, but from what I hear it's not good for high concurrency yet.
Most people that run local inference know about llama.cpp, vLLM, SGlang but very few know about this pure rust code inference engine named Atlas Inference dedicated exclusively to DGX Spark.
Made by a much smaller team than the regular engines we used to run, but with very promising performance.
Let's give the a shout-out!
github.com/Avarok-Cybersecur…
2
Bearly AI
