VRAM Sizing for LLMs: How Much GPU Memory Do You Need?

The most common workstation procurement mistake in AI is buying for the GPU name instead of the memory. For language models, VRAM is the specification that decides what you can run, at what precision, with how much context. Buy too little and models will not load; buy too much and capital sits idle. This guide gives you a working method to size GPU memory to your actual models — so you choose the right RTX 5090 or RTX PRO 6000 configuration the first time.

The four things that consume VRAM

GPU memory for inference is consumed by four things, and you have to account for all of them: the model weights, the KV cache that grows with context length and batch size, activations during the forward pass, and framework and CUDA overhead. People size for the weights alone and then run out of memory the moment they use a long prompt. The KV cache in particular scales with context — a long-context workload can need as much memory for the cache as for a small model's weights.

A quick rule of thumb for weights

Start with the model weights, which depend on parameter count and precision. The rough memory for weights is parameters multiplied by bytes per parameter: 2 bytes for FP16, 1 byte for INT8, roughly half a byte for 4-bit quantization. So a 13B model is about 26GB in FP16, 13GB in INT8, and roughly 7GB at 4-bit. A 70B model is about 140GB in FP16 but roughly 35GB at 4-bit — which is why quantization is what makes large models fit on a single card. Then add overhead for the KV cache and activations on top.

Mapping models to real configurations

• RTX 5090 (32GB GDDR7): 7B–13B models at FP16, or quantized models up to ~70B with a usable context window. The default for development and most local inference.
• Dual RTX 5090 (64GB combined): larger quantized models, longer contexts, or two workloads at once via model and data parallelism.
• RTX PRO 6000 (96GB ECC): the largest models a workstation can hold in full or near-full precision, in a single contiguous pool, with ECC reliability.

Fine-tuning needs more headroom than inference

Training and fine-tuning add optimizer states and gradients on top of the weights, which can multiply memory needs several times over full-precision inference. Parameter-efficient methods like LoRA and QLoRA dramatically reduce this — which is exactly why a 32GB RTX 5090 can fine-tune models that it could never train from scratch. If your plan includes fine-tuning, size for the training footprint of your chosen method, not the inference footprint.

Buy with headroom, not aspiration

Leave realistic headroom for the models you will actually run in the next year, plus growth in context length and batch size. But avoid 'aspirational' headroom — capacity for a workload you might run someday that sits unused in the meantime. A right-sized card you keep busy beats an oversized one you cannot justify.

Nexus Compute helps teams translate model lists and context requirements into the right VRAM tier — single 5090, dual 5090, or RTX PRO 6000 — then configures, tests, and warranty-backs the system and returns a quote within 48 business hours.