How much VRAM an LLM really uses

The question I get most often about running models locally is always the same: will it fit on my card?

And almost everyone does the same wrong math. Take the parameter count, multiply by the quantization bits, divide by eight, and conclude that a 32B model in Q4 takes 20 GB — so it fits comfortably on a 24 GB card.

Then they load the model and the GPU runs out of memory at 8 thousand tokens of context.

The math isn’t the problem. The problem is that it answers only a third of the question. The memory a model occupies has three parts, and the second one grows with the size of the conversation.

Part 1: the weights

This is the easy part, and the only one most people calculate. Each parameter takes a number of bits that depends on the quantization:

Quantization	bits/weight	bytes/weight
FP16 (unquantized)	16.0	2.00
Q8_0	8.5	1.06
Q6_K	6.6	0.82
Q5_K_M	5.7	0.71
Q4_K_M	4.9	0.61

Note that Q4_K_M is not 4 bits per weight, but ~4.9. K-quants store the sensitive layers (attention, embeddings) at higher precision than the name suggests. Anyone calculating with 4.0 underestimates the model by almost 20%.

Multiplied across the most common sizes:

Model	FP16	Q8_0	Q6_K	Q4_K_M
8B	16.1	8.5	6.6	4.9
12B	24.4	13.0	10.1	7.5
32B	65.6	34.9	27.1	20.1
70B	141.2	75.0	58.2	43.2

This is where the number everyone repeats comes from: 32B in Q4 = 20 GB. It’s correct. And it’s incomplete.

Part 2: the KV cache — the one nobody calculates

Every token that enters the conversation leaves a trace in memory. The model stores the key and value vectors for each token, in each layer, so it doesn’t have to recompute attention from scratch for every word it generates. That’s the KV cache, and it grows linearly with context.

The formula:

bytes per token = 2 × layers × kv_heads × head_dim × bytes_per_element

The 2 is because there are two matrices: key and value. You read the other three numbers straight from the model’s config.json — num_hidden_layers, num_key_value_heads and head_dim (or hidden_size / num_attention_heads, when head_dim isn’t declared).

For Llama 3 8B — 32 layers, 8 KV heads, head dimension 128, in FP16:

2 × 32 × 8 × 128 × 2 bytes = 131,072 bytes = 128 KiB per token

128 KiB per token sounds like nothing. Multiply it by a real context window:

Layers (typical model)	KiB/token	4k	8k	32k	128k
32 (≈8B)	128	0.54	1.07	4.29	17.18
40 (≈12B)	160	0.67	1.34	5.37	21.47
64 (≈32B)	256	1.07	2.15	8.59	34.36
80 (≈70B)	320	1.34	2.68	10.74	42.95

A 32B with 32k of context reserves 8.6 GB of cache alone — nearly half of what the weights take. At 128k, the cache exceeds 34 GB and becomes the most expensive item on the list, larger than the model itself.

Part 3: runtime overhead

That leaves the part that’s annoying to estimate, because it depends on the runtime, the driver and how much scratch space the attention kernel needs:

• CUDA context: between 300 and 600 MB, just for a process to exist on the GPU.
• Compute buffers: scratch space for attention and feed-forward, typically a few hundred MB, proportional to batch and context.
• Allocator fragmentation: available memory is never 100% usable.

In practice, 1 GB is an honest reserve for single-user inference on llama.cpp or Ollama. Servers running vLLM or TensorRT-LLM reserve considerably more, because they pre-allocate the entire cache at startup.

The complete calculation

Putting the three parts together for the case I care about — a 32B in Q4_K_M with 32k of context:

It fits — but with 4 GiB of headroom, not the 12 GB the quick math promised. And if you raise the context to 64k, it no longer fits.

How to verify it for real

All this arithmetic is for predicting. To actually know, measure — before and after loading the model, with the same context window you’ll really use:

# Before loading: what is already occupied (compositor, browser, etc.)
nvidia-smi --query-gpu=memory.used,memory.total --format=csv

# Load with the real context window, not the default one
OLLAMA_CONTEXT_LENGTH=32768 ollama run qwen3:32b "ok"

# How much the model actually reserved, and whether anything spilled to CPU
ollama ps

ollama ps shows a PROCESSOR column. If it says something like 70%/30% CPU/GPU, part of the model went to system RAM — and throughput drops by a factor of ten, because every token has to cross the PCIe bus.

When it doesn’t fit

In order of cost-benefit, from what hurts least to what hurts most:

1. Reduce the context. It’s the cheapest lever and the most ignored. Nobody needs 128k for what they do 90% of the time, and every halving gives back half the cache.
2. Quantize the KV cache. --cache-type-k q8_0 --cache-type-v q8_0 in llama.cpp halves the cache with a quality loss that rarely shows up in normal use. It’s the best trade on this list.
3. Drop one quantization step. Going from Q6_K to Q4_K_M gives back about 25% of the weights. The degradation is real, but usually smaller than intuition suggests.
4. Offload layers to the CPU. It works, almost anything fits — and it’s ten times slower. Last resort.
5. Switch models. A well-chosen 12B running entirely on the GPU delivers more value per second than a 32B running half on the CPU.

The order matters: the first two options cost almost nothing in quality and solve most cases. The fourth is the one everyone tries first, and it’s the worst.

---

In the next post I measure what this calculation predicts, on the bench, with tokens per second for each configuration. Arithmetic tells you what fits; only measurement tells you what’s worth it.