GPU Direct RDMA: How NVLink and Network Fabric Work Together

GPU Direct RDMA (GDR) is the technology that allows network interface cards to read from and write to GPU memory directly, bypassing both the CPU and system memory. Combined with NVLink for intra-node GPU-to-GPU communication, GDR enables a distributed training architecture where data flows from GPU memory on one node directly into GPU memory on another node without touching CPU caches — dramatically reducing latency and freeing CPU cycles for compute scheduling rather than data movement.

NVLink: The Intra-Node Fabric

NVLink 4.0 in the H100 SXM platform provides 900GB/s of total GPU-to-GPU bandwidth within a single node across the NVSwitch fabric. Eight H100 SXM GPUs connected via NVSwitch form a fully-connected all-to-all topology where any GPU can communicate with any other at full bandwidth simultaneously. This intra-node bandwidth dwarfs any inter-node network: 900GB/s on NVSwitch versus 50–100GB/s aggregate inter-node bandwidth on a typical 400GbE or NDR InfiniBand connection. The implication is significant: algorithm design should maximize intra-node data reuse before crossing the inter-node network boundary.

GPU Direct RDMA Architecture

GDR requires the NIC to be physically connected to the same PCIe root complex as the GPU it is serving, enabling peer-to-peer PCIe memory access. On DGX H100 systems, eight ConnectX-7 NICs are each attached to the same PCIe segment as a dedicated pair of GPUs, ensuring sub-2-microsecond NIC-to-GPU memory access. When NCCL initiates an inter-node all-reduce, GDR allows the NIC to read directly from the sending GPU's memory and write directly to the receiving GPU's memory — the CPU issues the NIC operation descriptor but does not participate in the data path.

• GDR requires nvidia-peermem kernel module loaded on all compute nodes
• NIC must be on the same PCIe domain as the target GPU — verify with nvidia-smi topo -m
• NCCL automatically detects and uses GDR when nvidia-peermem is available
• GDR performance degrades if GPU memory is not registered with the NIC (pinned memory required)
• PCIe P2P must be enabled in BIOS/UEFI — disabled by default on some server platforms
• Test with perftest tools (ib_write_bw with --use_cuda) to verify GDR is active and performing

Inter-Node Collective Optimization

NCCL's hierarchical collective algorithms exploit the NVLink/GDR architecture automatically when properly configured. For an all-reduce across 64 GPUs (8 nodes), NCCL first reduces across the 8 GPUs within each node using NVLink (intra-node reduce-scatter), then performs a ring-reduce across nodes using GDR over the inter-node network fabric (inter-node all-reduce), then broadcasts results back within each node over NVLink (intra-node all-gather). The inter-node ring operation is where network fabric latency and bandwidth dominate — it is the limiting factor that InfiniBand and 400GbE configurations are optimizing.

The NVLink-to-network boundary is where cluster performance is most sensitive to design decisions. A well-tuned GDR + InfiniBand or GDR + RoCE configuration can keep GPUs at 90%+ utilization during collective operations. A misconfigured one keeps them waiting.

How Nexus Compute Helps

Nexus Compute configures GPU servers with GDR enabled and validated out of the box. Our H100 SXM and H100 PCIe configurations include BIOS settings, PCIe topology validation, nvidia-peermem integration, and NCCL baseline benchmarks. We provide commissioning documentation that verifies the full data path from GPU memory through NIC to fabric, ensuring GDR is active before the system enters production. Contact us to discuss your multi-node GPU cluster configuration requirements.