Memory Bandwidth Is the Real Bottleneck in LLM Inference

Teams buying GPU servers for inference often compare TFLOPS first and memory bandwidth second. For large-language-model serving that ordering is backwards. The dominant phase of autoregressive inference — generating tokens one at a time — is limited by how fast the GPU can read weights and the KV cache out of memory, not by how fast it can multiply. Understanding this changes which GPU you buy and how you size a server.

Two phases, two bottlenecks

LLM inference has a prefill phase and a decode phase. Prefill processes the prompt in parallel and is compute-heavy, so it stresses the tensor cores. Decode generates output tokens sequentially; for each token the GPU must stream the entire set of model weights it needs, plus the growing KV cache, from HBM. At the batch sizes typical of interactive serving, decode is firmly memory-bandwidth bound. The faster the memory, the more tokens per second — almost linearly, until something else gives.

Why H200 and B200 outrun their FLOPS on paper

The H200 raised bandwidth to roughly 4.8 TB/s over the H100's 3.35 TB/s with the same compute engines, and in real decode-bound serving it delivers materially higher throughput as a direct result. Blackwell's B200 pushes bandwidth to roughly 8 TB/s with its dual-die HBM3e design. When you see a GPU post large inference gains without a matching jump in headline FLOPS, memory bandwidth is almost always the reason.

The KV cache eats your capacity

Bandwidth sets your speed; capacity sets your scale. The KV cache grows with batch size and context length, and it lives in HBM alongside the weights. Run out of room and you must shrink batches or context, which lowers throughput and utilization. This is why the H200's 141GB and the B200's 192GB matter for serving as much as for model size: more memory means larger batches and longer contexts before you hit a wall.

• Decode-heavy, interactive serving: prioritize memory bandwidth and capacity over raw FLOPS.
• Long-context workloads: budget HBM explicitly for a KV cache that scales with context length.
• High-concurrency serving: capacity headroom enables bigger batches and better GPU utilization.
• FP4 on Blackwell: lower-precision weights cut bandwidth demand per token and stretch effective throughput.

Spec the server to the bottleneck

Once you accept that serving is bandwidth-bound, the configuration follows. Choose the GPU with the bandwidth and capacity your workload needs, then ensure the rest of the node does not reintroduce a bottleneck: NVLink and NVSwitch so multi-GPU models are not throttled by inter-GPU links, enough system memory and NVMe to stage models quickly, and cooling that prevents thermal throttling under sustained load. A bandwidth-rich GPU in a starved chassis gives back the advantage you paid for.

Nexus Compute sizes inference servers to the bottleneck that actually governs your tokens per second, then builds, tests, and warranty-backs the result. Share your model and concurrency targets and we will return a validated configuration and quote within 48 business hours.