🎉 Meet Hy3 from @TencentHunyuan, a 295B MoE with 21B active params and 256K context. Day-0 support is now live in SGLang! 1️⃣ Agent-first: rivals flagship open-source models with 2–5x the parameters across reasoning & agentic tasks 2️⃣ Anti-hallucination training: hallucination rate cut from 12.5% to 5.4% in real-world evals 3️⃣ Multi-turn intent tracking: MRCR jumps from 42.9% to 75.1% 4️⃣ MTP EAGLE speculative decoding out of the box, with an FP8 checkpoint (Hy3-FP8) for cost-effective serving Cookbook: docs.sglang.io/cookbook/auto… Run it now with SGLang!

🐱 LongCat-2.0 is now fully open-source — MIT licensed, no restrictions. Since our launch a few days ago, the response from the community has been incredible. Thank you for all the feedback, discussions, and interest. Today, we’re releasing the model weights and inference code to everyone. ◆ 1.6T MoE · ~48B active · 1M token context ◆ Agent-native: Integrates directly with Claude Code, OpenClaw, and Hermes Agent ◆ Deployment: Support both GPU and NPU platforms— verified on large-scale domestic clusters 📑 Tech Blog: longcat.ai/blog/longcat-2.0/ 🤗 HuggingFace: huggingface.co/meituan-longc… 💻 GitHub: github.com/meituan-longcat/L… 🪄 ModelScope: modelscope.ai/collections/me… 👇 Inference Code GPU: github.com/sgl-project/sglan… NPU: github.com/meituan-longcat/S…

🚀 New blog: Agent-Assisted SGLang Development, the story of how we turn benchmarking, profiling, and kernel optimization know-how into executable agent skills. Agent-assisted workflows are saving our team massive engineering hours while delivering major gains across the stack: ⚡️ 71.4% throughput & TTFT 456→168ms for Qwen3-Next via allreduce fusion ⚡️ 29–49% TTFT reduction on long-context prompts via router tokenization deduplication ⚡️ Up to 2.32x diffusion denoising speedup via Spectral Progressive Diffusion ⚡️ 10 B200 kernel tasks at 1.13x–2.75x speedups via KDA-Pilot; 3 PRs merged upstream ⚡️ 1.41x faster LTX-2 VAE decode, saving 9.7 GiB peak memory And rigor is built into every step: benchmarks are fixed before any patching, baseline and candidate share the same ABI, and every change must be backed by profile evidence, eliminating benchmark reward hacking. Each iteration passes a Humanize/RLCR review loop before proceeding. Read the full blog to see how we're rethinking development workflow 👇

Today we’re releasing Laguna XS 2.1. It’s a small upgrade to the Laguna XS.2 model, the same 33B total / 3B active MoE and stronger results on multilingual coding and terminal-style tasks. Available now on @huggingface, @OpenRouter, and via Poolside API.

Built on PyTorch, Ray, SGLang, and NVIDIA Megatron-LM, Miles is an open source framework from RadixArk for large-scale LLM reinforcement learning post-training. Miles uses PyTorch for models, numerics, profiling, and extensibility; Ray for orchestration; SGLang for rollout generation; and Megatron-LM for distributed training. The framework supports asynchronous rollout and training, NCCL/RDMA weight synchronization, MoE-aware rollout/training alignment, low-precision recipes, LoRA, fault tolerance, observability, and extension points for custom algorithms and model architectures. 🔗 Read more in our latest blog from the Miles Team: pytorch.org/blog/miles-a-pyt…

NVIDIA inference software keeps driving down token costs, long after AI infrastructure is deployed. ⚡ In just one month on NVIDIA Blackwell, software optimizations improved DeepSeek V4 performance by up to 5×, reducing token costs to roughly one-fifth of previous levels. NVIDIA's integrated inference software stack compounds improvements across runtimes, kernels, networking, and hardware, delivering up to 20× higher throughput on the same GPU. Co-designed with NVIDIA GPUs, CPUs, networking, and systems, and powered by CUDA-native open source frameworks, NVIDIA's inference software stack ensures new model breakthroughs and optimizations run on NVIDIA from day zero, and keep improving throughput and lowering cost after deployment. See how @Baseten, @Cognition, @DeepInfra, @togethercompute, and @Cursor_ai are turning continuous software innovation into lower cost per token: nvda.ws/4eRT43m

🚀 New blog: Improving DeepEP MoE Load Balance in SGLang with Waterfill and LPLB We're introducing two dispatch-time load balancers for DeepEP MoE. Even with EPLB, a single batch still hits ranks unevenly. Waterfill and LPLB fix that residual imbalance at runtime, no change to model semantics. 1️⃣ Waterfill for the dense shared expert Pours shared-expert work onto lighter ranks (“filling the valleys”) instead of every rank paying it locally. Near-zero overhead via shared-expert fusion into the DeepEP layout. ⚡️ 1.48% to 4.66% on DeepSeek V3/R1 across MMLU, GPQA, GSM8K ⚡️ V4 Flash: 49,253 → 51,677 tok/s ( 4.92%) 2️⃣ LPLB for redundant routed-expert replicas EPLB splits hot experts evenly, but live traffic drifts from calibration. LPLB solves a per-layer min-max LP on-GPU each batch to split replica traffic optimally and shrink the busiest rank. ⚡️ 0.84% to 7.34%, strongest when redundant replicas exist (red16/red32) Both preserve accuracy: same logical top-k, identical replica weights, only the physical rank changes. Huge thanks to the @nvidia team for the collaboration!

[HF ML Club India] We are proud to announce the second IRL event from the HF ML Club India in Bengaluru. This time we have parterned with the @RedHat_AI India team (particularly their PyTorch Engg team). It is happening on the 25th of July. Talks: > @adithya_s_k speaks on OpenEnv > @adarshxs talks about SGLang > I cover torch profiling > Mansi (RedHat) takes on torch distributed > Arkadip (RedHat) provides in-depth knowledge about gpu2gpu comms in distributed setups This is also golden opportunity for folks who were not accepted for the first event, to come and enjoy the talks and network with peers.

🎉 Meet LFM2.5-230M from @liquidai, their smallest model yet at 230M params, but it punches way above its weight. Day 0 Support is live on SGLang! Built on the LFM2 architecture for on-device deployment: > Blazing-fast inference, runs everywhere, from cloud GPUs to low-cost CPUs > Capable of tool use and structured data extraction > Outperforms models twice its size Try it now on SGLang!

It is not too late to _actually_ own your inference. Introducing: Modal Auto Endpoints.

It is not too late to _actually_ own your inference. Introducing: Modal Auto Endpoints.

Increase inference performance by up to 15x without sacrificing responsiveness. DFlash, an open source lightweight block diffusion model designed for speculative decoding, delivers up to 15x higher throughput on NVIDIA Blackwell while maintaining the same user interactivity target. Instead of drafting tokens one at a time, it proposes a whole block in a single pass for the main model to verify in parallel. Adoption is drop-in with support in @lmsysorg SGLang, TensorRT-LLM, and @vllm_project.