Steve Hou retweeted
This thread is awesome but very technical for a generalist like me. Here's a cheat sheet from GPT as you go through as I needed one... 🤣
I agree with this post whole heartedly but I’d push it even further. The interconnect IS the binding constraint for AI even more so than memory. If we want faster inference & training with better economics we are best served by designing our interconnect first and then working backwards towards the optimal chip architecture.
Today’s chips weren’t really designed with this principle in mind. There is no better example than running autoregressive decode on a GPU. Despite all those reticle sized logic dies & CoWoS integration decode runs at under 20% of peak FLOPs on Blackwell, wasting silicon and burning power while waiting for memory. The naive solution has always been to increase memory bandwidth whether that’s adding more HBM or using SRAM. However, that is a vast simplification of the problem which I’ll explain later. But if you were clever you’d have realized while reading that you could feed those idle FLOPs by streaming weights over the interconnect itself. Wallah 🪄 you just discovered the idea that forms the basis behind disaggregated memory from first principles.
But sadly this currently doesn’t work on Nvidia’s hardware. NVLink5 carries 1.8 TB/s against 8 TB/s of local HBM, and scale out is 80x behind that. The “pipe” is smaller than the memory at the other end and thus leads to worst token/sec if its relied upon. But we get an interesting lemma out of this which is that remote memory is only as fast as the interconnect. Therefore you must balance the pipe for the memory it attaches to. SRAM needs an 80 TB/s link, HBM needs 2 TB/s, and LPDDR gets away with a couple hundred GB/s. So Nvidia selling a rack of 72 GPUs, each GPU’s memory is pretty segregated. The core idea is still sound though but this raises a question, why would Nvidia build a fabric that’s high bandwidth and high latency leading to memory access being segregated per GPU?
It’s because they were optimizing for training over inference. Training is dominated by collectives on huge tensors, and a couple microseconds of latency on a huge all reduce operation is just noise so the bandwidth gains justify the latency tradeoff. But more importantly, this also works because it matches what the chip is good at. GPUs are great at hiding latency with occupancy (also what allows them to be OK for training) but bandwidth is the only thing warp switching can’t create. You can justify a 224G PAM4 FEC with overhead when you have a chip that’s designed to be latency tolerant as well. It’s a latency tolerant fabric for a latency tolerant chip. Maybe a good design for training but inference inverts this completely.
Now everyone knows decode is bandwidth bound so you might assume again that more/higher BW memory and thus higher BW interconnects are necessary. However, it’s the exact opposite and the name of the game is actually lower latency and that’s why despite having high bandwidth memory MFU on decode is still so low and also why I made the point earlier that the interconnect is MORE important than the memory itself as well as the chip architecture. In part two I will explain why lower latency interconnects are not just ideal for inference but also allow you to get away with a smaller cheaper memory and a simpler chip architecture.
1
2
19
2,929
