Network Bandwidth Requirements for Large-Scale LLM Training

Infrastructure teams planning network fabrics for large-scale LLM training frequently approach bandwidth requirements empirically — deploying a cluster, observing throughput, and iterating. A more rigorous approach derives network bandwidth requirements analytically from model parameters, parallelism strategy, and optimizer state transfer volumes. This analysis allows proper fabric sizing before hardware procurement and avoids the costly scenario of discovering network bottlenecks after a GPU cluster is in production.

Parallelism Strategies and Their Network Implications

Modern LLM training uses three parallelism dimensions with different network requirements. Data parallelism all-reduce exchanges gradient tensors proportional to model parameter count at each training step: a 70B parameter model requires transferring 140GB of gradients (FP16) per all-reduce across all data-parallel ranks. Tensor parallelism (intra-layer parallelism) requires high-bandwidth, low-latency all-reduce within each tensor-parallel group at every transformer layer — this is the most latency-sensitive communication pattern and benefits most from InfiniBand or high-quality RoCE. Pipeline parallelism sends activation tensors between pipeline stages — bandwidth requirement is proportional to batch size and hidden dimension, typically lower than gradient all-reduce.

Quantifying All-Reduce Bandwidth for Common LLM Sizes

For data-parallel gradient all-reduce, the ring-based algorithm transfers 2*(N-1)/N * parameter_bytes per step across all ranks, where N is the data-parallel degree. A 70B parameter model in BF16 with 128 data-parallel ranks requires approximately 109GB of gradient data transferred per step. At a 1-second step time target (batch size and GPU compute dependent), the all-reduce must complete in under 100–200ms, requiring sustained inter-node bandwidth of 500–1,000GB/s across the full cluster. For 128 nodes with 8 NICs each at 400GbE, available aggregate bandwidth is 128 * 8 * 50GB/s = 51,200GB/s — but the ring topology uses only 2x inter-node bandwidth per node, so effective throughput is bounded by the ring hop bandwidth, not aggregate.

• 7B model, 8 GPUs data parallel: all-reduce ~14GB gradients, completes in <10ms on 400GbE
• 13B model, 16 GPUs: ~26GB gradient all-reduce, network rarely a bottleneck
• 70B model, 64 GPUs: ~109GB gradient all-reduce, network latency begins to matter significantly
• 175B model, 256 GPUs: pipeline + tensor parallelism required; tensor-parallel bandwidth is critical
• Tensor parallelism degree 8: all-reduce at every transformer layer, sub-millisecond latency required
• ZeRO-3 data parallelism: parameter scatter/gather adds 3x communication volume versus standard DP

Tensor Parallelism: The Latency-Critical Case

Tensor parallelism splits transformer layer weight matrices across multiple GPUs, requiring an all-reduce at every layer forward pass and every layer backward pass. For a 70B model with 80 transformer layers and tensor-parallel degree 8, this is 160 all-reduce operations per training step on the critical path. If each all-reduce requires 500 microseconds (a reasonable 400GbE estimate for medium message sizes), tensor parallelism communication adds 80ms per step — roughly 8% of a 1-second step time. On InfiniBand with 200 microsecond all-reduce latency, this overhead drops to 32ms. For models where compute time per step is shorter, this differential is proportionally more significant.

Tensor parallelism makes your training throughput directly proportional to your network latency. It is the parallelism strategy that most clearly justifies InfiniBand over Ethernet for large-model training at scale.

How Nexus Compute Helps

Nexus Compute provides network bandwidth modeling as part of our cluster design service. Given your target model architecture, parallelism strategy, and training throughput requirements, we calculate inter-node communication volumes and recommend fabric specifications that eliminate network bottlenecks. Our configurations have been validated with Megatron-LM and DeepSpeed workloads across model sizes from 7B to 70B+ parameters. Contact us for a training infrastructure design consultation.