The 8GB VRAM fit table is lying by 7 percent

Best ollama model for 8GB VRAM: qwen2.5:7b (Q4_K_M) — about 5.4 GiB all-in at 8k context, the most headroom, and the only one of the five that still fits at 32k. llama3.1:8b (≈6.2 GiB) and qwen3:8b (≈6.4 GiB) fit at 8k. gemma2:9b needs ≈8.6 GiB at 8k and fails — 4k context only.

What does 8GB VRAM actually hold?

Claim: an "8 GB" card never holds 8 decimal GB of model — it holds 8 binary GiB, and the desktop takes its cut before the model arrives. Marketing bought the "8" twice and gave it two different meanings; your out-of-memory killer only honours one of them. The card's 8 GB is 8 × 2³⁰ bytes = 8.59 decimal GB. A GGUF file's 4.92 GB is 4.92 × 10⁹ bytes. Same word, two rulers, a fixed 7.4% gap between them — I've started calling it the seven-percent ghost, because it haunts every VRAM table on the internet and almost no table confesses which ruler it picked.

The second cut nobody budgets: the card is never empty. A Linux desktop with a compositor parks 0.3–0.5 GiB there, the CUDA/ROCm context takes ~0.3 more. Headless, you get ~7.6 GiB to play with; on a desktop, plan on 7.3 GiB usable. Every fit call in this piece is judged against that line.

Why do VRAM tables disagree by 7 percent?

Claim: the tables aren't rounding differently — they're mixing rulers, and at the margin the mix decides the answer. Take gemma2:9b. The file lists 5.76 GB (decimal). A naive table compares 5.76 against "8" and prints 72% full — fine, ship it. The honest stack at 4k context: 5.37 GiB (file) + 1.31 GiB (KV) + 0.6 GiB (overhead floor) = 7.28 GiB against 7.3 usable. Same model, same card, same context: the table's 72% and reality's 99.7% are the same row. The division is one line — GiB = GB ÷ 1.074 — and every fit verdict below shows its division.

Best ollama model for 8GB VRAM: the verdict table

Claim: at 8k context, four of the five community defaults fit — and the one that doesn't is the 9B the listicles keep recommending. File sizes measured today from the Hugging Face file API (bartowski GGUF repos, Q4_K_M — the ollama default quant for these families; mistral:7b ships q4_0 by default, within ~2% of the measured file):

curl -s https://huggingface.co/api/models/bartowski/Meta-Llama-3.1-8B-Instruct-GGUF/tree/main \
  | jq '.[] | select(.path | test("Q4_K_M"))'

verify: Q4_K_M.gguf → 4,920,739,232 bytes, pulled 2026-11-01. Same one-liner ran for all five repos.

model (ollama tag)	file, decimal GB	÷ 1.074 = GiB	KV @ 8k	overhead floor	total GiB	8k verdict
qwen2.5:7b	4.68	4.36	0.44	0.6	5.40	fits — roomiest; holds 32k (6.71)
mistral:7b	4.37	4.07	1.00	0.6	5.67	fits, roomy
llama3.1:8b	4.92	4.58	1.00	0.6	6.18	fits; 16k hairline (7.18); 32k fails (9.18)
qwen3:8b	5.03	4.68	1.13	0.6	6.41	fits — watch the thinking budget
gemma2:9b	5.76	5.37	2.63	0.6	8.59	fails 8k — 4k only (7.28, hairline)

The winner is boring on purpose: ollama pull qwen2.5:7b gets you the most usable context per GiB, and it is the only row here that still fits when you double the context to 32k (6.71 GiB). The fail row is the receipt that justifies the piece: gemma2:9b misses 8k by 1.29 GiB even with zero overhead budgeted — 5.37 + 2.63 = 8.0 against 7.3 usable. No quant of it fixes that; Q4_K_M is already the small one.

How much VRAM does the KV cache eat?

Claim: context is the second model you're loading, and architecture decides the price — per token, gemma2:9b charges six times what qwen2.5:7b does. The KV cache is the model's working memory for the conversation: 2 bytes × layers × kv-heads × head-dim, at fp16, per token. Qwen2.5-7B's slim 28×4×128 attention charges 56 KiB/token. Gemma-2-9B's 42×8×256 charges 336 KiB/token. That per-token rate, not the file size, is what kills long context on a small card.

model	KV @ 4k	KV @ 8k	KV @ 16k	KV @ 32k
qwen2.5:7b (56 KiB/tok)	0.22 GiB	0.44	0.88	1.75
mistral:7b / llama3.1:8b (128)	0.50	1.00	2.00	4.00
qwen3:8b (144)	0.56	1.13	2.25	4.50
gemma2:9b (336)	1.31	2.63	5.25	10.50

One 2026-specific footnote: qwen3 is a thinking model. Its reasoning tokens are context tokens — a short answer that "thought" for 3,000 tokens pays for those 3,000 at 144 KiB each until the context resets. On a 7.3 GiB card the thinking budget is not a vibe setting; it is the KV table above, compounding.

The arithmetic, line by line

Usable line: 8 GiB − 0.4 (desktop, compositor + display) − 0.3 (CUDA context) ≈ 7.3 GiB; headless recovers ~0.3 more. Overhead floor 0.6 = CUDA context ~0.3 + compute buffer ~0.3 at 8k; the honest range is 0.4–1.2 GiB depending on backend and context — the gemma2 fail verdict holds across the entire range, which is why the floor is enough. KV rates: llama3.1-8B 32L×8kv×128d; qwen2.5-7B 28×4×128; qwen3-8B 36×8×128; gemma2-9B 42×8×256; mistral-7B-v0.3 32×8×128 — all ×2 bytes ×2 (K and V). GiB conversions: file bytes ÷ 2³⁰ (4,920,739,232 → 4.58; 4,683,074,240 → 4.36; 4,372,812,000 → 4.07; 5,027,784,224 → 4.68; 5,761,057,728 → 5.37). gemma2 honesty note: half its layers are sliding-window, so llama.cpp's real KV is smaller than this conservative full-attention figure — the fail verdict does not need the discount to hold. last tested 2026-11-01.

What this table cannot tell you

Claim: every number above is arithmetic on public sizes — and the two numbers the table still guesses are overhead and MoE, which is exactly where the next measurement lives. The overhead floor is a range wearing a uniform; any calculator that prints one overhead number is guessing, including our own GGUF VRAM calculator piece — its wound, the one that refuses MoE math, is documented there. And the fit line moves with your desktop: boot headless, recover ~0.3 GiB, and gemma2:9b at 4k goes from hairline to merely tight. What would settle it: the same five models through ollama on a real 8 GB card, nvidia-smi numbers printed against this table's predictions. That bench is not this piece — the card on this desk has no business pretending to be one.

last tested 2026-11-01 — file sizes pulled live from the Hugging Face file API; the fit math is division, not vibes.