Network Monitoring and Observability for GPU Cluster Fabrics

A GPU cluster that is underperforming due to network issues typically shows no obvious alerts in traditional monitoring systems. Interface utilization may look healthy. Packet loss may be zero. Switch CPU is idle. Yet training throughput is 30% below expected MFU because intermittent PFC pause storms are stalling collective operations, or because ECMP hash collisions are creating hot links that saturate while other paths sit idle. Effective observability for GPU cluster fabrics requires a purpose-built telemetry stack that captures the metrics that actually matter for AI workload performance.

Critical Metrics for AI Cluster Network Observability

Standard SNMP interface counters (ifInOctets, ifOutOctets, ifInErrors) are necessary but not sufficient. For AI cluster fabrics, the essential additional metrics include: PFC pause frame counts per-priority-class per-port (high PFC rates indicate congestion or misconfiguration), ECN congestion marking rates (quantify how frequently the congestion control algorithm is active), RDMA error counters from NIC hardware (packet sequence number errors, remote access errors, timeout counters), per-flow latency histograms from switch hardware telemetry, and NCCL collective operation timing from the application layer. These metrics must be correlated in time to diagnose causal relationships.

Streaming Telemetry: Beyond SNMP Polling

SNMP polling at 60-second intervals misses sub-second events that determine AI cluster performance. Modern network switches support gNMI/gRPC streaming telemetry that pushes counter values at 1–10 second intervals or event-driven notifications on threshold crossing. Arista EOS, Cisco NX-OS, and Juniper Junos all support gNMI streaming. The telemetry stream feeds into a time-series database (InfluxDB, Prometheus with GNMI exporter, or vendor-specific platforms like Arista CloudVision) where per-second PFC event rates and ECN marking rates can be correlated against application-layer training throughput metrics.

• gNMI streaming at 10-second intervals for interface counters; 1-second for PFC and ECN counters
• RDMA NIC counters: query via ethtool -S or mlxlink tool, scrape with custom Prometheus exporter
• InfiniBand: NVIDIA UFM provides subnet-level telemetry including per-port error counters and flow rates
• NCCL profiling: NCCL_DEBUG=INFO and NCCL timeline with Nsight Systems captures collective timing
• Prometheus + Grafana: standard observability stack; maintain GPU metrics (DCGM exporter) alongside network
• Alert thresholds: PFC pause >1% of port time, ECN marking >5% of packets, any RDMA sequence errors

Correlating Network Events with Training Performance

The most valuable observability capability is temporal correlation between network events and training step time. Instrument your training loop to emit step start and stop timestamps with step index to a metrics endpoint. Graph these against per-second PFC pause counts and ECN marking rates from the network telemetry stream. When training step time spikes coincide with PFC pause events on specific ports, the diagnosis is straightforward: congestion on those links is causing pause propagation that stalls collective operations. Without this correlation, the same symptom appears as unexplained training throughput variance and consumes days of debugging time.

The moment you can overlay training step time on a graph with per-second PFC pause counts from your switch telemetry, you have transformed AI cluster network debugging from guesswork into a diagnostic discipline.

Alerting and Runbook Integration

Effective alerting for GPU cluster networks requires threshold calibration against your specific workload. A cluster running all-reduce every 1 second will have different baseline PFC rates than one running pipeline-parallel training with less frequent collective operations. Establish baselines during a known-good training run and alert on deviation rather than absolute thresholds. Pair each alert with a runbook: PFC storm alert triggers a flow that checks DCQCN configuration and ECN threshold settings; RDMA error rate alert triggers NIC firmware version check and cable optical power measurement. Runbook-backed alerts reduce mean time to resolution from hours to minutes.

How Nexus Compute Helps

Nexus Compute provides a reference observability stack for GPU cluster networks including gNMI telemetry configuration for supported switch platforms, Prometheus exporter configurations for RDMA NIC counters, pre-built Grafana dashboards correlating network and GPU metrics, and alert rules calibrated for AI training workloads. We include observability stack setup in our cluster commissioning service and provide runbooks for the most common network performance failure modes. Contact us to learn more about our cluster deployment and observability services.