How Much VRAM Do You Need to Run an LLM? Model Size Guide

How much VRAM you need for an LLM comes down to one calculation: parameters × bytes per weight, plus room for the KV cache and runtime overhead. That works out to about 2 GB per billion parameters at 16-bit, 1 GB at 8-bit and 0.5–0.6 GB at 4-bit, with 10–30% on top. A 70B model at 4-bit therefore needs roughly 42–50 GB.

The VRAM Formula: Weights, KV Cache and Overhead

The GPU memory an LLM uses is the sum of three parts, and the first one dominates. You can do this LLM VRAM calculator in your head:

• Weights: parameters × bytes per weight. A 16-bit format (FP16 or BF16) uses 2 bytes, an 8-bit format (INT8 or FP8) uses 1 byte, and 4-bit formats land at about 0.5–0.6 bytes once the stored scale factors are counted.
• KV cache: the attention state kept for every token already processed. It grows with context length and with the number of concurrent requests (see the KV cache section below).
• Overhead: the CUDA context, activation buffers and allocator fragmentation. Budget 10–30% on top of the weights as a rule of thumb, not a specification.

VRAM (GB) ≈ parameters (billions) × bytes per weight + KV cache (GB) + overhead
Quick estimate: total ≈ weights × 1.1 to 1.3

Worked example: a 70B model at 4-bit has 70 × 0.55 = 38.5 GB of weights. Multiply by 1.1–1.3 and you get about 42–50 GB. That is a tight fit on a single 48 GB card at short context and does not fit on a 24 GB card at all. Use this same arithmetic to size the VRAM for an LLM before you rent or buy a GPU.

A unit note: GB here means 10^9 bytes, while nvidia-smi reports MiB (2^20 bytes), so expect differences of up to about 7%.

LLM VRAM Requirements by Model Size and Quantization

The table applies the formula to common parameter counts. Pick your model size, pick a precision, and read off the approximate weight memory. These are weights only, so the KV cache and overhead come on top.

Quantization is the lever that matters most. Storing each weight in 8 bits instead of 16 halves the weights, and 4 bits cuts them to roughly a quarter. The trade-off is accuracy: the loss is usually small at 8-bit and easier to notice at 4-bit and below, so test a quantized model on your own prompts before committing. Common 4-bit formats include GGUF (used by llama.cpp), AWQ and GPTQ. LLM quantization is the most economical way to fit a bigger model on the same card.

How Context Length Changes VRAM Use (KV Cache Explained)

The KV cache stores the key and value vectors of every token the model has already read, in every layer, so it does not recompute them for each new token. It grows linearly with context length, scales with the number of concurrent sequences, and depends on the architecture. Here is the formula:

KV cache bytes = 2 × layers × kv_heads × head_dim × context_tokens × bytes_per_value
(the 2 is one key plus one value)

As a worked example, take Qwen3-32B, whose official config lists 64 layers, 8 KV heads and a head dimension of 128. In BF16 that is 2 × 64 × 8 × 128 × 2 bytes = 262,144 bytes, or 256 KiB per token.

Qwen3-32B’s config lists 40,960 positions natively, so the 128k row shows how the cache scales rather than a supported default. The cache is not a rounding error: at 128k the BF16 cache alone is about 34 GB, roughly half the 65.5 GB of BF16 weights.

Two things keep the cache manageable. Grouped-query attention (GQA) lets many query heads share a few KV heads; Qwen3-32B has 64 attention heads but only 8 KV heads, so its cache is 8× smaller than full multi-head attention would need. Second, you can store the cache itself at lower precision: llama.cpp has cache-type options for the K and V caches (including q8_0), and vLLM documents an FP8 KV cache with the –kv-cache-dtype fp8 option. An 8-bit cache is half the size of a 16-bit one.

Mixture-of-Experts Models: Why Active Parameters Do Not Set VRAM

A mixture-of-experts (MoE) model routes each token through only a few expert sub-networks, but any token can be routed to any expert, so all experts must sit in memory. Active parameters set compute per token; total parameters set the memory you need.

gpt-oss-120b is the clearest example. Its model card lists 117B parameters with 5.1B active per token, and its config shows 128 experts with 4 used per token. The MoE weights were post-trained with MXFP4 quantization, and the card says the model fits on a single 80 GB GPU. The checkpoint shards add up to about 65 GB, close to what the formula predicts for a roughly 4-bit model. Stored in FP16, 117B parameters would take about 234 GB.

Qwen3-30B-A3B-Instruct-2507 shows the same pattern at a smaller scale: 30.5B total parameters, 3.3B activated, 128 experts with 8 active. It needs memory like a 30B dense model (about 61 GB in BF16, about 17 GB at 4-bit) while computing per token like a model closer to 3B. gpt-oss-120b also has a small KV cache for its size: its config shows 36 layers, 8 KV heads, a head dimension of 64, and every second layer limited to a 128-token sliding window. How much of that saving you see depends on the runtime.

Which Models Fit on 24 GB, 48 GB and 96 GB of VRAM?

If you searched for how much VRAM for LLM use, the practical answer is a tier. The table below lists models whose sizes come from their official cards or repositories, at the precision that makes them fit.

Three patterns stand out. A 24 GB card is a 4-bit card for anything around 30B parameters, with gpt-oss-20b as the comfortable exception. A 48 GB card runs a 32B model at 8-bit with real context. Only at 96 GB do you keep a 32B model in BF16 or load gpt-oss-120b with room to spare.

Choosing a GPU VPS: What Fits in 96 GB of VRAM

Contabo’s GPU VPS has one dedicated NVIDIA RTX PRO 6000 Blackwell Server Edition with 96 GB of GDDR7, alongside 18 vCPUs, 96 GB of system RAM and 900 GB of NVMe storage. According to Contabo’s launch post, it costs €999 per month excluding VAT. Our GPU VPS launch post has the full specification, and the GPU VPS pricing comparison puts the price next to other providers. Applying the tables above, three configurations make sense on this card:

• One large model at higher precision: Qwen3-32B in BF16 (65.5 GB) or gpt-oss-120b (about 65 GB) leaves roughly 30 GB for the KV cache and overhead.
• A mid-size model with long context: Qwen3-32B at 8-bit (about 33 GB) plus a 128k BF16 KV cache (about 34 GB) totals roughly 67 GB, with about 29 GB to spare. Mind the model’s native context limit.
• Several small models side by side: four 16 GB-class models such as gpt-oss-20b take about 64 GB, leaving roughly 30 GB for their caches.

For a GPU server for AI work, the dedicated 96 GB matters more than the vCPU count: weights that fit entirely in GPU memory avoid the slow CPU offload described below.

When VRAM Runs Out: Offloading, Quantization and Multi-GPU Options

If the model does not fit, work down this list in order:

1. Lower the precision. Moving from 16-bit to 8-bit, or 8-bit to 4-bit, is the biggest single saving. This is where LLM quantization pays off first.
2. Shorten the context and quantize the KV cache. Cutting the context window or storing the cache in 8 bits frees memory without touching the weights.
3. Offload layers to the CPU and system RAM. llama.cpp lets you choose how many layers live on the GPU with its -ngl option and runs the rest on the CPU. It works, but expect a large speed penalty because system RAM is much slower than GPU memory. For CPU-friendly runtimes, see our Llamafile guide and the KoboldCpp guide.
4. Split the model across several GPUs with tensor parallelism. It adds memory but needs a fast link between cards to scale well.

How to Check VRAM Use on Your Server

Estimates are a starting point, so measure with nvidia-smi. Watch memory.used climb as the weights load and again when the server allocates its KV cache:

nvidia-smi
nvidia-smi –query-gpu=memory.used,memory.total –format=csv -l 1
nvidia-smi –query-compute-apps=pid,process_name,used_memory –format=csv

The first command gives a snapshot, the second logs memory every second while a model loads, and the third lists memory per process. Some inference servers reserve a large share of GPU memory up front, so memory.used can look high even on an idle server; compare it with what you calculated rather than treating it as the minimum. If you run containers, set up the GPU first, then track usage over time. For an end-to-end walkthrough, read our guide to LLM inference on a GPU VPS.

FAQ: LLM VRAM Requirements

Disclaimer: Product specifications, features, and prices mentioned in this article are subject to change and may vary by region, billing term, and active promotions. Please check each provider’s or brand’s official website for current figures, pricing, and local currency rates.