Storage Area Networks (SAN) for Multi-GPU Clusters: Design and Sizing

A storage area network (SAN) provides block-level storage to multiple hosts over a dedicated high-speed network fabric, separate from the general-purpose LAN. For multi-GPU AI clusters, a well-designed SAN delivers high-throughput shared access to training datasets, checkpoint volumes, and model repositories — allowing storage capacity to be managed and expanded centrally rather than distributed across compute nodes. But a poorly designed SAN becomes the most expensive bottleneck in your GPU cluster, sitting between hundreds of thousands of dollars of compute hardware and the data it needs to process. SAN design for AI requires discipline and specificity.

Fibre Channel vs iSCSI vs NVMe-oF: Fabric Protocol Selection

Traditional SAN deployments use Fibre Channel (FC) at 32 Gb/s or 64 Gb/s per port as the storage network protocol. FC is mature, low-latency, and operationally proven at large scale. iSCSI runs SCSI block protocol over standard Ethernet, allowing SAN deployment without dedicated FC infrastructure — acceptable for smaller deployments but limited in latency performance compared to FC or NVMe-oF. NVMe over Fabrics (NVMe-oF) running over InfiniBand (RDMA) or RoCE Ethernet represents the current state of the art for AI storage networks. NVMe-oF eliminates the SCSI translation layer entirely, reducing latency to near-local-NVMe levels and supporting the queue depth and parallelism that GPU clusters demand. For new AI SAN deployments, NVMe-oF over InfiniBand or 100/200GbE RoCE is the recommended protocol.

SAN Sizing Methodology

Begin with the compute-side demand: calculate the aggregate storage throughput required by all GPU nodes operating at full data loading utilization. Add 25 percent headroom for checkpoint write traffic. This number is your minimum required aggregate SAN throughput. Work backward from this figure through the storage controller tier (each controller has a maximum front-end bandwidth) and then through the fabric switch tier (each ISL must be sized to avoid oversubscription under worst-case traffic). The ratio of host-side ports to ISL ports — the oversubscription ratio — should not exceed 2:1 for AI workloads. A 4:1 oversubscription ratio, common in enterprise SAN deployments for file servers, will cause fabric congestion under GPU cluster I/O patterns.

• Calculate aggregate GPU data consumption in GB/s before sizing any SAN component.
• Use non-blocking or low-oversubscription SAN switch fabrics (2:1 maximum recommended for AI workloads).
• Size storage controller front-end bandwidth to match or exceed aggregate host demand — controller bottlenecks are common in under-designed SAN architectures.
• Deploy at least two paths between each host and the storage system (multipathing) for both redundancy and bandwidth aggregation.
• Separate training dataset volumes from checkpoint volumes on different controller ports to prevent checkpoint write storms from affecting data loader reads.
• Plan for storage controller and fabric switch expansion without disruptive reconfiguration — AI storage requirements grow quickly.

Host Bus Adapter Selection and Configuration

Each GPU server in the cluster connects to the SAN via a host bus adapter (HBA) or converged network adapter (CNA). For Fibre Channel SANs, 32GFC or 64GFC HBAs from Marvell (Cavium) or Broadcom are standard. For NVMe-oF over InfiniBand, the same Mellanox/NVIDIA ConnectX adapters used for GPU-to-GPU communication serve dual duty as storage fabric ports, eliminating the need for a separate HBA. This dual-use model reduces PCIe slot consumption in GPU servers, which is already highly contested. Ensure that InfiniBand fabric bandwidth is partitioned or reserved appropriately between GPU communication and storage traffic — uncontrolled competition between these two traffic types will degrade both.

A SAN designed for general enterprise use will fail in an AI cluster environment. AI I/O patterns require a SAN designed explicitly for high-throughput, low-oversubscription, concurrent access.

How Nexus Compute Helps

Nexus Compute designs and delivers complete SAN architectures for multi-GPU AI clusters. Our solutions architects perform detailed bandwidth and latency modeling for your specific GPU cluster configuration, then specify the correct storage controllers, fabric switches, and host adapters. We deploy and validate the full fabric before delivery and provide integration support to ensure your parallel file system or block storage volumes are accessible at full rated bandwidth from every node in your cluster.