I agree with this post whole heartedly but I’d push it even further. The interconnect IS the binding constraint for AI even more so than memory. If we want faster inference & training with better economics we are best served by designing our interconnect first and then working backwards towards the optimal chip architecture.
Today’s chips weren’t really designed with this principle in mind. There is no better example than running autoregressive decode on a GPU. Despite all those reticle sized logic dies & CoWoS integration decode runs at under 20% of peak FLOPs on Blackwell, wasting silicon and burning power while waiting for memory. The naive solution has always been to increase memory bandwidth whether that’s adding more HBM or using SRAM. However, that is a vast simplification of the problem which I’ll explain later. But if you were clever you’d have realized while reading that you could feed those idle FLOPs by streaming weights over the interconnect itself. Wallah 🪄 you just discovered the idea that forms the basis behind disaggregated memory from first principles.
But sadly this currently doesn’t work on Nvidia’s hardware. NVLink5 carries 1.8 TB/s against 8 TB/s of local HBM, and scale out is 80x behind that. The “pipe” is smaller than the memory at the other end and thus leads to worst token/sec if its relied upon. But we get an interesting lemma out of this which is that remote memory is only as fast as the interconnect. Therefore you must balance the pipe for the memory it attaches to. SRAM needs an 80 TB/s link, HBM needs 2 TB/s, and LPDDR gets away with a couple hundred GB/s. So Nvidia selling a rack of 72 GPUs, each GPU’s memory is pretty segregated. The core idea is still sound though but this raises a question, why would Nvidia build a fabric that’s high bandwidth and high latency leading to memory access being segregated per GPU?
It’s because they were optimizing for training over inference. Training is dominated by collectives on huge tensors, and a couple microseconds of latency on a huge all reduce operation is just noise so the bandwidth gains justify the latency tradeoff. But more importantly, this also works because it matches what the chip is good at. GPUs are great at hiding latency with occupancy (also what allows them to be OK for training) but bandwidth is the only thing warp switching can’t create. You can justify a 224G PAM4 FEC with overhead when you have a chip that’s designed to be latency tolerant as well. It’s a latency tolerant fabric for a latency tolerant chip. Maybe a good design for training but inference inverts this completely.
Now everyone knows decode is bandwidth bound so you might assume again that more/higher BW memory and thus higher BW interconnects are necessary. However, it’s the exact opposite and the name of the game is actually lower latency and that’s why despite having high bandwidth memory MFU on decode is still so low and also why I made the point earlier that the interconnect is MORE important than the memory itself as well as the chip architecture. In part two I will explain why lower latency interconnects are not just ideal for inference but also allow you to get away with a smaller cheaper memory and a simpler chip architecture.
chips get all the love but the interconnects across all levels from c2c to rack-to-rack are as important, and many chip makers are sleeping on it until very recently. Even most interconnects people just want it to be as transparent as possible, just send and receive the bits with lower error rates and lower energy per bit, wrong long-term direction imo. Interconnects are part of the living creature, so many things happening in your blood vessels in addition to just moving stuff, and your axons do much more than carrying spikes.
People do not appreciate interconnects, smaller volume, poor margin, messy ecosystem, manual process, it's been a spiral of grinding, and it is largely invisible. How often do you see people tearing down transceivers and die shot of DSP chips vs logic chips? How often do you see high res pictures of all the connector's gold fingers on the NVL72 cartridge? Because it sounds boring, it's just making contacts, shoving electrons and photons, what a simple problem.
But that is deceiving, and theres so much to it. You might want 576 to begin with, had to cut down to 288, then to 144, and finally to 72, and that barely worked first time. You are entering the domain of analog and mixed signaling, you are fighting copper real estate with power delivery, you are getting impedance mismatch and reflections and interference at every stupid interfaces, your optical components' and connectors' backreflection is making your laser mode hop.. And we are not even going into the thermal and strain-stress, the reliability of how many times you can actually mate your connector, the horrible jobs people are doing across the stack from science-project-originated photonic PDKs to hand cleaved laser dies to optical engines to rack manufacturing, on spec-ing out the requirements, the tolerances.. On top of all these, people thinking about where the bits should be going and people who know what the bits have to go thru are two totally different groups of people.
But it is shifting, pluggable volumes shipped are doubling and tripling for scale-out, scale-up domain asks for much higher bw than scale-out, and interconnects are inevitable even if you cram as much compute and memory onto a single wafer. People will see it always has been interconnects, the chips people have already been doing it on the chips, that you can sort it out with your chip designers and foundries, and now you need to work with more people to sort it out from chips to boards/trays to racks to pods to data halls and data centers. These people speak very different languages and care about very different things, and it will take a lot of effort to pull order out of all the chaos.
At the end of the day, it is such a crazy problem to work on, such a beautiful thing to make, millions of amps of current flipping 1e20 of flops, sextillions of photons carrying thousands of terabits per second, a few tons of copper, tens of thousands of fibers totally few hundred kilometers, one scale-up domain. You absolutely need a group of people that appreciate the beauty and care about the craft behind the grinding to make it together.
