How Many GPUs Does an AI Server Need?

The question arrives in almost every specification we are sent: four GPUs or eight? It is usually asked as a budget question. It is really a workload question. The number of GPUs in a server is not something you choose and then justify afterwards. It falls out of two facts about your work — whether a single job has to span several GPUs, and how much memory that job needs.

One job across many GPUs, or many jobs across one each

This is the fork in the road, and everything else follows from it. If your largest job fits on a single GPU — most production inference, most LoRA fine-tuning of mid-size models, most classical machine learning — then a multi-GPU server is a scheduling convenience. Each card runs its own work. The GPUs barely need to talk to each other, and PCIe between them is entirely adequate.

If your largest job has to be split across GPUs, the picture changes. A model too big for one card, or a training run you want to finish this month rather than next quarter, means the GPUs exchange gradients or activations at every step. Now the fabric between them is part of the compute, and the count you can usefully choose is constrained by what that fabric supports.

Teams that get this wrong usually buy the right number of GPUs on the wrong interconnect. Eight PCIe cards in a chassis is not the same machine as eight SXM GPUs on an HGX baseboard, even where the GPU silicon is identical.

Start with memory, not with GPU count

Work out the memory footprint first and the GPU count often answers itself. For inference in BF16, a working rule of thumb is two bytes per parameter for the weights, plus the KV cache, which grows with context length and the number of concurrent requests. Full fine-tuning is far heavier. Weights, gradients, an FP32 master copy and two optimiser moments come to roughly sixteen bytes per parameter with Adam, before activations are counted at all.

Set that against real GPU memory. An H100 SXM gives you 80GB. An H200 gives 141GB, a B200 180GB, an AMD Instinct MI300X 192GB. An L40S gives 48GB, and an A100 either 40GB or 80GB depending on the part. A job needing 300GB of resident state is a multi-GPU job on any of them. A job needing 60GB is a single-GPU job on most.

That arithmetic gives you three outcomes, and each points at a different machine.

• If the job fits on one GPU with room for the batch size you actually use, choose the count from how many jobs run at once.
• If it fits only across two or four GPUs, you need a real NVLink domain of at least that width.
• If it needs eight or more, you are buying an HGX-class node, and possibly more than one.

Why eight-GPU nodes exist

Eight is not a marketing number. It is the width of the HGX baseboard the industry standardised on, and the systems built around it — the Supermicro SYS-821GE-TNHR, the Dell PowerEdge XE9680, the HPE Cray XD670, the Lenovo ThinkSystem SR685a V3 — are variations on the same eight-GPU module. On that board NVSwitch connects every GPU to every other at full NVLink bandwidth. There is no near pair and far pair. Any GPU reaches any other at the same speed.

That matters because collective operations — the all-reduce and all-gather primitives underneath data and tensor parallelism — run at the pace of the slowest link in the group. A uniform fabric makes your parallelism strategy a software decision rather than a topology puzzle. It is also why parallel degrees come in powers of two: two, four and eight divide cleanly into an eight-GPU node, and eight-GPU nodes divide cleanly into a cluster.

The eight-GPU node has become the unit of account for the industry. Reference architectures, InfiniBand rail designs, scheduler defaults and framework tuning all assume it. Buying the standard shape buys the accumulated tuning that came with it.

The honest case for four

Four GPUs is a serious configuration, not a compromise, and for a good number of teams it is the correct one.

• Every job you run today fits within four GPUs, and nothing on your roadmap changes that within the planning cycle.
• Your work is inference and interactive development, where GPUs are handed to people rather than pooled into one large run.
• Your rack cannot supply the power or remove the heat of a dense eight-GPU chassis, and changing that is an electrical project rather than a purchase order.
• You would rather add a second node next year than fill an eight-GPU baseboard with cards nobody is using yet.

Four-way platforms — the Dell PowerEdge R760xa, the four-way H200 NVL systems, which bridge four PCIe cards into one NVLink domain — also arrive in ordinary 2U and 4U chassis with conventional air cooling. They fit racks that already exist. That practical fact settles more procurement decisions than any performance argument.

When four is a false economy

The case against starting small is simple. You cannot add GPUs to a baseboard that was never there. An eight-GPU HGX node is a different chassis, a different power delivery design and a different cooling design from a four-GPU PCIe server. If you buy four now and need eight next year, you do not upgrade. You buy a second machine, and you now have two nodes synchronising over a network instead of one node synchronising over NVLink.

That is fine when the work is embarrassingly parallel and expensive when it is not. Splitting a single training run across two four-GPU nodes puts the network in the critical path of every step. It means InfiniBand or high-speed Ethernet with RDMA, adapters, cabling and a switch — none of which you needed when the same eight GPUs sat on one baseboard.

There is a quieter cost as well. Chassis, CPUs, system memory, boot storage and the support contract are much the same whether you populate four GPU slots or eight. Spread across four GPUs, that overhead is a larger share of what you paid. Density is what amortises it.

The workload signals that settle it

• A model larger than one GPU's memory, with tensor parallelism across the whole node — eight GPUs on NVSwitch.
• Data-parallel training of a model that fits on one GPU — any count works; more GPUs simply finish sooner.
• Production inference serving many concurrent requests — count driven by throughput and redundancy rather than fabric, and MIG partitioning on A100, H100 and later parts can lift utilisation further.
• A shared research server for a team of engineers — four is often enough, because the constraint is people, not parallelism.
• A model that only just fits on one GPU today — assume it will not next year, and buy the fabric now.

Power decides this more often than anyone plans for

An H100 SXM is rated to 700W. Eight of them, together with two server CPUs, a large complement of system memory, network adapters and fans, put a single node into a class of power draw that many enterprise racks were never provisioned for. The GPU count you can install is frequently larger than the count your rack can actually run.

Check three numbers before the configuration is settled: the circuit and PDU capacity feeding the rack, the heat the room can remove, and the depth and weight the rack will take. Where the answer falls short, direct liquid cooling is the way through — liquid-cooled eight-GPU nodes exist precisely because air cooling ran out first. But liquid cooling is a facilities commitment, not a line item you tick on an order form.

A sensible way to decide

Size the largest single job you must run, in gigabytes of resident state, then leave headroom for the batch size and context length you actually use. Divide by the memory of the GPU you intend to buy and round up to a power of two. That is the minimum NVLink domain you need. Then count how many such jobs run concurrently at your busiest hour and multiply. If the result is four or fewer and stays there through a planning cycle, buy the four-GPU node and be content with it. If it reaches eight, buy the eight-GPU node now. The fabric is the part you cannot retrofit.

If your model already fits, the extra density buys headroom you may not use this year. If it does not fit, no amount of scheduling discipline will make four GPUs behave like eight.

How Nexus Compute helps

As an independent procurement partner, we help you turn a four-versus-eight GPU decision into a concrete, validated configuration — sourced through authorized channels and quoted within 48 business hours. Our specialists configure first and quote second, so what you receive actually works on day one.