LLM Development Workstation

It depends on the GPU configuration and quantization. We size the VRAM to the models you intend to run and advise on what is realistic for full-precision versus quantized inference. The arithmetic is straightforward: roughly two bytes per parameter at BF16, roughly half a byte at 4-bit, plus a KV cache that scales with context length and concurrency. A 70B model is therefore out of reach unquantized on any workstation but comfortable at 4-bit on either configuration here. Send us the exact model, target context length and expected concurrency and we will confirm before you order.

Two cards suit two jobs: one model served while another is fine-tuned, or two engineers each pinned to their own GPU. One large card suits one big job, because a 96GB pool runs models that a 64GB split cannot, with no tensor-parallel configuration and no inter-GPU communication cost. If your work is mostly a single large model with long contexts, take the single card. If it is several medium models in parallel, take the two. We will make the call with you against your actual model list rather than in the abstract.

You have three options and all of them cost something. Quantize further, which reduces memory and usually costs some output quality. Offload layers to system RAM, which works but drops throughput sharply because inference then moves at the speed of the PCIe link rather than the card. Or move to hardware with more VRAM. Reducing batch size and context length also helps and is often the cheapest first move. We would rather size the machine correctly at the start than have you discover this during an evaluation deadline.

We commonly pre-configure vLLM, Ollama, and the Hugging Face stack, but we install whatever your team standardizes on. Versions are pinned and documented, which matters here more than elsewhere, because inference stacks move quickly and a working configuration is worth being able to reproduce. If you intend to deploy to a server later, tell us which stack production will run so the workstation matches it rather than diverging from it.

For development and staging, yes: a locally served endpoint on the office network is a normal use of this machine. For production it is the wrong shape. A workstation has no redundant power, no hot-swap parts and no out-of-band management, so an internal service that people depend on ends up being a single point of failure sat under someone's desk. Once real users depend on the endpoint, move it to /products/gpu-servers and keep the workstation for development.

When you need to serve models to many concurrent users in production, or train at a scale beyond a single machine. We can advise on the transition to our GPU Server line. Concurrency is usually the trigger that arrives first: every additional simultaneous request adds KV cache, so a machine that comfortably serves one or two developers can run out of memory at ten users on the same model. If your roadmap has a user count on it, tell us and we will size for where that leads rather than where you are now.