GGUF VRAM calculator: the three numbers that decide

How much VRAM a GGUF model needs = weights + KV cache + overhead. Weights: parameters × bits-per-weight ÷ 8, plus ~8% for the parts quantization doesn't shrink. KV cache: 2 × layers × kv_heads × 128 × 2 bytes × context. Overhead: 0.6–1.2 GiB. The gguf vram calculator at github.com/mrsaynothing/gguf-vram-calculator does this arithmetic for you — formulas printed, nothing hidden.

Four public GGUF files, four predictions from the math, and the worst miss was 0.04 GB — under one percent of the file. That receipt is the whole argument for doing this on paper before the loader does it with an out-of-memory banner.

Who this is for: anyone who has ever typed "will this quant fit" into a search box or a Discord at 1am. The words you need first — bpw is bits per weight, what a quant spends on one parameter; KV cache is the model's scratchpad for the conversation so far, and it grows with every token of context; overhead is the CUDA context, cuBLAS and compute buffers the GPU keeps for itself; a quant (Q4_K_M, Q8_0) is the file format that set the bpw.

How do you calculate GGUF VRAM?

Three lines, and you can audit every one of them:

weights  = params × 1e9 × (quant bpw ÷ 8) × 1.08
kv cache = 2 × layers × kv_heads × head_dim(128) × 2 bytes × context
overhead = CUDA context + cuBLAS + compute buffers (≈ 0.6–1.2 GiB)

verify: the weights line checked against four public GGUF files — Llama-2 7B Q4_K_M predicted 4.09 vs real 4.08 GB, Llama-3 8B Q4_K_M 4.88 vs 4.92, Mistral 7B Q4_K_M 4.39 vs 4.37, Llama-3 8B Q8_0 8.55 vs 8.54. Max drift 0.04 GB. Three file sizes re-pulled live from the Hugging Face API 2026-10-31 (Llama-2 row: the tool's own verified table — the repo API gated the re-check). Last tested 2026-10-31.

The 8% on the weights line is not fudge for fudge's sake: embeddings and the output head stay above the quant's nominal bits-per-weight in real files, and the factor soaks that up. A formula you can't explain is a horoscope; this one fits inside a tweet.

How accurate is the math, really?

Model · quant	Real file	This math	Drift
Llama-2 7B · Q4_K_M	4.08 GB	4.09	+0.01
Llama-3 8B · Q4_K_M	4.92 GB	4.88	−0.04
Mistral 7B · Q4_K_M	4.37 GB	4.39	+0.02
Llama-3 8B · Q8_0	8.54 GB	8.55	+0.01

One unit slip worth printing, because we made it ourselves: the tool's README called this column GiB, but model files list decimal GB — 4.92 GB is 4.58 GiB, and that 0.34 difference is a third of an 8 GB card. The card's monitor counts GiB; the file's size is quoted in GB. Now both do.

The drift never crosses 0.05 GB — under one percent, on files anyone can download and weigh. That is the gap between "should fit" and the red banner. The loader tax overhead band is the one honest guess in the set: it varies by driver and backend, which is why it stays a range and not a digit.

What about the KV cache — the number everyone forgets?

Here is the KV surprise. The cache is 2 bytes (fp16) per element, and the element count scales with context. Llama-3 8B has 32 layers and 8 KV heads at head_dim 128:

2 × 32 × 8 × 128 × 2 = 131,072 bytes/token = 128 KiB/token

 8k context  →  1.0 GiB
16k context  →  2.0 GiB
32k context  →  4.0 GiB

verify: arithmetic from the formula above; cross-checkable against the calculator's own output for the same model. Last tested 2026-10-31.

The file never grows. The bill does. That 4.9 GiB model on an 8 GB card is comfortable at 8k and impossible at 32k — same weights, same card, four more GiB of scratchpad. Every "it fit yesterday and OOMs today" thread is this table wearing a trench coat.

Why older 7B models are hungrier at context than newer ones

Llama-2 7B runs full multi-head attention: 32 KV heads, not 8. The per-token bill is 4× bigger — 512 KiB/token, a full GiB of cache per 2k tokens of context. Newer families (Llama-3, Mistral) use grouped-query attention and pay a quarter of that. When an old guide's context math feels insane, check the KV-head count before blaming your GPU.

How much VRAM for text-generation-webui?

The search box pairs this question with text-generation-webui often enough that it deserves its own answer. The UI loads GGUF through its llama.cpp loader, and the knobs you set there are exactly the three numbers above: n_ctx (context — feeds the KV cache line), n-gpu-layers (how much of the model lives on the card at all), and cache_type, where dropping the KV cache from fp16 to q8_0 halves that line at a small quality cost. Leave flash-attention on if the build offers it; the overhead band gets friendlier. The loader also keeps its own module and workspace memory on the card, which is part of why overhead stays a range.

One dry truth about loaders in general: they will happily begin loading a file the card cannot finish holding. The calculator exists so the allocator never gets to be the one who tells you.

What won't the math cover?

Three honest blind spots, printed on the tool itself: MoE routing (mixture-of-experts models shuffle experts in ways per-layer math doesn't see), draft models (speculative decoding parks a second model in the same memory), and your desktop's share — the browser is already holding VRAM before the model asks, and it has first claim. Your tabs are not in the formula. They are, however, in your task manager.

The calculator is one static page — no build, no network, open it from disk if you like: github.com/mrsaynothing/gguf-vram-calculator. The companion pieces live nearby: how to run GGUF models locally (/en/space/2026-09-07-how-to-run-gguf-models-locally) and the vLLM–GGUF border rules (/en/space/2026-10-09-vllm-gguf-faq). The open wound: MoE sharding is where this math goes blind, and mixture-of-experts models are exactly where the local-LLM world is heading. When the formula learns to see experts, the calculator gets that line too — until then, the 0.04 GB promise holds only for dense models, and the honest thing is to say so.

The behaviour behind the three numbers — what offload does when the model misses the card — is its own piece: the offload cliff, measured.