AI GPU Capacity Planning: How Many GPUs Do You Actually Need?

Capacity planning is where most AI infrastructure budgets are won or lost. Buy too little and your team queues for compute and falls behind; buy too much and you carry idle, depreciating GPUs that never earned their keep. The goal is not a round number of GPUs — it is a defensible count derived from the workloads you will actually run. This guide walks through how to get there.

Separate training, fine-tuning, and inference

These three workloads have completely different capacity profiles, and lumping them together is the most common planning error. Training from scratch is bursty and interconnect-heavy. Fine-tuning is shorter and often fits on a single node. Inference is steady-state, latency-sensitive, and scales with request volume rather than model size. Size each one separately, then add them up — never estimate a single blended 'AI workload.'

Start from memory, then time

The binding constraint for training is almost always GPU memory. A rough rule: full fine-tuning needs roughly 16-20 bytes per parameter once you account for weights, gradients, optimizer states, and activations. A 70B-parameter model in mixed precision therefore needs well over a terabyte of aggregate GPU memory, which maps to a single 8x H100 (80GB) or H200 (141GB) node before you add headroom. Once memory tells you the minimum node, training time tells you how many nodes you need to hit your iteration deadline.

A workable sizing sequence

• List each workload with its model size, precision, and how often it runs (daily, weekly, one-off).
• Compute the per-job GPU memory requirement and round up to the nearest node (4-GPU or 8-GPU).
• Estimate runtime per job, then multiply by job frequency to get GPU-hours per week.
• Divide weekly GPU-hours by the hours in a week to get the steady-state GPU count, then add 20-30% headroom for growth and failures.
• Reconcile against your interconnect: tightly-coupled training above one node needs InfiniBand, which changes the node and switch count.

Plan for utilization, not peak

A cluster sized for absolute peak demand sits idle most of the time. A cluster sized for the 80th percentile of demand, with a scheduler to queue the occasional spike, delivers far better return. Decide explicitly how much queueing your team will tolerate during peaks — that single policy choice can change your GPU count by 30% or more.

Right-sized capacity is not the largest cluster you can afford. It is the smallest cluster that keeps your team unblocked at the 80th percentile of demand.

Leave room to grow without a forklift

Models grow, teams grow, and new projects arrive mid-year. Choose a node and rack design you can extend by adding GPUs or nodes rather than replacing what you have. That means specifying network fabric, power, and cooling for the cluster you expect in 18 months, even if you populate it for today.

Nexus Compute turns a capacity model into a validated, warranty-backed configuration — sizing GPU count, memory, and fabric against your real workloads, testing the build before it ships, and returning a full quote within 48 business hours.