Redundancy and Failover Planning for On-Prem GPU Clusters

Cloud providers give you failover for free — a failed instance is replaced before you notice. On-prem gives you no such thing. A dead GPU or a failed node in your own cluster is your problem, on your timeline, and often at the worst possible moment. The question worth answering before you deploy, not after a failure, is how much spare capacity you actually need — not "as much as possible," but sized to what actually fails and what it actually costs you when it does.

What actually fails, and how often

GPUs themselves fail at a low but nonzero annual rate — thermal cycling and memory errors account for most of it. Power supplies and fans fail more often than GPUs do. In practice, network interface cards and cabling generate the largest share of support tickets, even though the part itself rarely dies outright — a flaky link is harder to diagnose than a dead component. The right way to think about failure rate is components × count × per-part failure rate, not "GPUs might fail sometimes." A 64-GPU cluster has 64 chances for a GPU failure, but many more chances for a PSU, DIMM, or NIC failure.

Training and inference need different redundancy

A training job that checkpoints regularly can resume on a different node after losing a few hours — the requirement is enough spare capacity to pick the job back up, not zero downtime. A production inference endpoint has no such grace period: a failed node means degraded service for real users right now. Before sizing spare capacity, be honest about which category actually dominates your cluster's use. Teams that size training-grade sparing for an inference-heavy cluster are routinely underprotected.

N+1 versus N+2: a rule of thumb by cluster size

• Small clusters (under ~8 GPUs): a fully idle spare node is rarely economical. A stocked parts kit — one or two spare GPUs, PSUs, DIMMs — plus a documented swap procedure covers most failures faster and cheaper.
• Mid-size clusters (roughly 8–64 GPUs): N+1 node-level sparing is the standard baseline — one spare node covers the statistically likely single failure.
• Large clusters (64+ GPUs): N+2 or a small dedicated spare pool. With more hardware in service, simultaneous failures become likely enough that a single spare can already be consumed mid-repair when the next failure hits.

Spare parts vs. a spare full node

A fully redundant idle node is expensive to buy and arguably wasteful to leave powered off. Most teams below the largest scale are better served by a stocked parts kit — PSUs, memory, NICs, and one or two spare GPUs — and a rehearsed swap procedure, reserving a full spare node for clusters large enough to justify it. This is exactly where lead times bite: sourcing the identical GPU SKU you deployed months ago, after a failure, can run into months of wait if you didn't plan for it at initial purchase.

Build the plan in before rollout, not after

Redundancy strategy retrofitted onto a cluster that's already live is more expensive and more disruptive than one planned from the start. Decide your sparing level, document the failover procedure, and identify who's on call before the first rack ships — and rehearse an actual failover once, deliberately, before you need it to work under pressure.

Nexus Compute configures spare-parts kits and sparing strategy alongside the primary cluster build, sourced through the same authorized channels and matched to your exact configuration — quoted within 48 business hours, so replacement hardware is a phone call away instead of a new lead-time clock.