Network Congestion Control for Distributed AI Training: ECN and DCQCN

Distributed AI training generates traffic patterns that are fundamentally different from any workload that traditional network congestion control was designed to handle. All-reduce collectives cause synchronized, incast-style traffic bursts where every GPU in a cluster transmits simultaneously to multiple destinations. Without precise congestion control, these bursts saturate switch queues, trigger PFC pause frames, and propagate backpressure across the entire fabric — degrading training throughput by 20–40% compared to a properly tuned environment.

ECN Mechanics in AI Cluster Networks

Explicit Congestion Notification (ECN) enables switches to signal congestion to endpoints without dropping packets. Switches mark the ECN bits in packet headers (CE codepoint) when queue occupancy exceeds a configured threshold. For RoCE v2 traffic, the destination NIC generates a Congestion Notification Packet (CNP) back to the source upon receiving an ECN-marked packet. The source then reduces its transmission rate. The key parameters are the ECN minimum threshold (when to start marking) and maximum threshold (when to mark all packets). Setting these thresholds correctly requires understanding your switch's buffer architecture and your workload's traffic burst characteristics.

DCQCN Parameter Tuning

DCQCN (Data Center Quantized Congestion Notification) is the algorithm implemented in Mellanox/NVIDIA ConnectX NICs that responds to CNP signals. When a CNP is received, the rate-limiter reduces the transmission rate by a multiplicative factor (alpha). Over time, if no further CNPs arrive, the rate increases additively (RAI phase) until a byte counter threshold is reached, after which it increases rapidly (HAI phase). The default DCQCN parameters in ConnectX NICs are tuned for general workloads and are frequently suboptimal for all-reduce collective patterns. Critical tuning parameters include CNP interval, initial alpha value, and the G (gain) coefficient for rate-limiter updates.

• ECN minimum threshold: start at 50KB for 100GbE, 150KB for 400GbE port buffers
• ECN maximum threshold: 3–5x minimum threshold for probabilistic marking
• CNP interval: 1–4 microseconds; shorter intervals react faster but increase CPU overhead
• DCQCN alpha (initial rate reduction): 0.0625 is a reasonable starting point
• Byte counter threshold for HAI phase: tune based on message size distribution
• Verify CNP generation and reception counters in ethtool/mlxlink statistics

Incast Mitigation Strategies

Incast — where many senders transmit to a single receiver simultaneously — is structurally unavoidable in all-reduce operations. Beyond DCQCN tuning, additional mitigation strategies include: increasing ToR switch buffer allocation for RoCE priority queues, using tree-based collective algorithms (ring-reduce to binary-tree) that reduce simultaneous sender count, enabling SHARP in-network reduction on InfiniBand fabrics to perform reductions in-switch, and configuring NCCL (NVIDIA Collective Communications Library) chunk size to match the network's optimal message size for the collective algorithm in use.

The single most effective tuning action in most misperforming RoCE clusters is correcting the ECN threshold mismatch between switch buffer size and NIC-side response parameters. Default configurations are almost never optimal for AI collective traffic.

How Nexus Compute Helps

Nexus Compute provides validated DCQCN and ECN configurations for each switch and NIC combination in our portfolio. Our cluster commissioning service includes collective throughput benchmarking using NCCL tests, with iterative tuning until throughput targets are met. We document all configuration parameters for your operations team and provide runbooks for common congestion-related failure scenarios. Contact us to discuss network performance optimization for your AI training infrastructure.