NVLink and NVSwitch on H100: Why the GPU Fabric Decides Scaling

Buyers fixate on GPU count and memory, but for multi-GPU training the interconnect between the GPUs is just as decisive. On the H100, that interconnect is NVLink, switched by on-board NVSwitch chips. It is the reason an 8-GPU HGX node can behave like one large accelerator instead of eight cards fighting over a shared bus. If you train or fine-tune large models, this is the part of the spec sheet to read most carefully.

What NVLink does on the H100

NVLink is NVIDIA's high-bandwidth, low-latency GPU-to-GPU link. The fourth-generation NVLink on the H100 SXM5 provides up to 900 GB/s of total bidirectional bandwidth per GPU — roughly an order of magnitude more than a PCIe Gen5 x16 path. That bandwidth is what allows the gradients, activations, and parameters exchanged during distributed training to move fast enough that the GPUs are not left waiting on each other.

NVSwitch turns point-to-point links into an all-to-all fabric

NVLink on its own connects GPUs in pairs or small groups. NVSwitch is the crossbar that turns those links into a fully non-blocking fabric. On an 8-GPU HGX H100 baseboard, four NVSwitch chips ensure every GPU can reach every other GPU at the full NVLink bandwidth simultaneously. That all-to-all property is exactly what collective operations like all-reduce need, and it is why 8-way tensor parallelism scales cleanly on HGX.

Why it matters more than raw GPU count

• Tensor parallelism splits a single layer across GPUs and exchanges data every step — it is bandwidth-hungry and lives or dies on NVLink.
• Collective operations (all-reduce, all-gather) run at the speed of the slowest link in the fabric.
• PCIe-only H100 systems share a far narrower path, so the same job scales sub-linearly as GPU count rises.
• Adding GPUs without adequate fabric bandwidth buys you idle silicon, not throughput.

For tightly-coupled training, the fabric between the GPUs determines how much of your GPU investment actually turns into throughput.

Scaling beyond one node

Within a node, NVSwitch handles GPU-to-GPU traffic. Across nodes, you move to a network fabric — typically InfiniBand (NDR 400Gb/s class) or high-speed Ethernet with RoCE, often with GPUDirect RDMA so GPUs exchange data without involving the host CPU. The NVLink Switch System can also extend NVLink across a small number of nodes for the most bandwidth-sensitive deployments. Either way, the principle holds: design the fabric to match the GPUs, or the GPUs sit idle.

Nexus Compute designs the full interconnect — NVSwitch within the node and InfiniBand or RoCE across nodes — as part of every H100 server and cluster we configure, then tests and warranty-backs the system and quotes within 48 business hours.