H100 Training Throughput: What Drives Real-World Performance

An H100 is rated for enormous peak compute, but the number on the datasheet is not the throughput you get on a real training run. The gap between peak and delivered performance is where money is won or lost. Understanding the levers — precision, the Transformer Engine, memory bandwidth, and interconnect — lets you size a server that actually delivers, rather than one that looks fast on paper.

Precision is the biggest single lever

The H100's Hopper architecture introduced FP8 alongside the Transformer Engine, which dynamically manages precision across a network's layers. For transformer training and inference, FP8 can roughly double effective throughput over BF16 while preserving accuracy when applied correctly. The headline tensor-core figures — into the petaFLOP range with sparsity — are FP8 numbers. If your framework and model are set up to use FP8 through the Transformer Engine, you unlock a large share of the H100's advantage over the prior generation; if not, you are leaving performance on the table.

Memory bandwidth keeps the tensor cores fed

Compute throughput only matters if data reaches the cores fast enough. The SXM5 H100's HBM3 delivers about 3.35 TB/s of memory bandwidth, versus roughly 2 TB/s on the PCIe HBM2e card. Many training and inference kernels are memory-bound, not compute-bound, which is why the SXM5 part can outrun the PCIe part by more than their FLOP ratings alone would suggest. Bandwidth, not just capacity, is part of the throughput equation.

Interconnect determines multi-GPU scaling

• Single-GPU throughput is set by precision and memory bandwidth.
• Multi-GPU throughput is capped by the NVLink/NVSwitch fabric once GPUs must synchronize.
• On HGX, 900 GB/s NVLink keeps all-reduce cheap, so scaling stays close to linear.
• On PCIe-only systems, communication overhead grows with GPU count and scaling falls off.

The parts of the system that quietly cap throughput

Even a perfect GPU configuration can be throttled by the rest of the node. Storage that cannot stream training data fast enough stalls the GPUs between batches. Too few CPU cores or too little system memory bottlenecks data loading and preprocessing. Inadequate cooling forces the GPUs to throttle their clocks under sustained load. Real throughput is a property of the whole machine, which is why a balanced node beats a GPU-heavy, everything-else-starved one.

Sizing for sustained, not peak, performance

Datasheet peaks assume ideal conditions. Plan around sustained throughput under your real workload, with cooling rated for continuous 700W-per-GPU operation, storage and CPU sized to keep the GPUs fed, and an interconnect matched to how your jobs parallelize. That is how you turn rated FLOPs into finished training runs.

Nexus Compute configures and stress-tests H100 servers as balanced systems — GPUs, memory bandwidth, storage, CPU, and cooling together — so the throughput you specify is the throughput you receive, with a quote returned within 48 business hours.