1 post
Posts tagged quantization
-
How much VRAM an LLM really uses
The back-of-the-envelope math — parameters times bits — is off by several gigabytes, because it ignores the KV cache. Here is the full calculation, with the formula, the numbers and how to verify it on your own GPU.