KV cache quantization — the bill your context hides

What is KV cache quantization? Storing the attention cache — the K and V tensors every past token contributes — in a smaller numeric format (q8_0 or q4_0) instead of 16-bit floats, so long contexts cost proportionally less memory. Measured on llama.cpp b12069, Qwen3-4B Q4_K_M: at 32k context the f16 cache bills 4608 MiB against a 2382 MiB model — q8_0 cuts that to 2448, q4_0 to 1296. The weights are a one-time purchase; the cache is the meter running.

Download the weights once — the context invoices every session.

What is KV cache quantization?

Claim: KV cache quantization lowers the numeric precision of the cache only — the K/V tensors — and never touches the weights on disk. Picture a hotel. The weights are the building: bought once, the same every night. Every token of context checks in as a guest, and K and V are its luggage — K is where that token's meaning lives, V is the meaning itself. Quantizing the cache is charging the guests for a smaller room: same guest, thinner luggage.

• K ("key") — the address of a past token's meaning: how relevant is token 400 to the word I just read?
• V ("value") — the meaning itself, delivered when the address matches.
• f16 / q8_0 / q4_0 — 16-bit, 8.5-bit and 4.5-bit storage. In llama.cpp the flag is -ctk/-ctv; in Ollama it's the OLLAMA_KV_CACHE_TYPE env var (public record — we did not re-bench Ollama here).

How much memory does the KV cache use?

Claim: per token, this model's f16 cache costs 144 KiB — 36 layers × 8 KV heads × 128 head_dim × 2 (K+V) × 2 bytes — and the loader's own boot lines print it to the byte. Two lines from the receipt, unedited:

llama_kv_cache: size = 4608.00 MiB ( 32768 cells,  36 layers,  1/1 seqs), K (f16): 2304.00 MiB, V (f16): 2304.00 MiB
llama_kv_cache: size = 1296.00 MiB ( 32768 cells,  36 layers,  1/1 seqs), K (q4_0):  648.00 MiB, V (q4_0):  648.00 MiB

verify: those are the size = lines llama.cpp prints at boot — captured, not transcribed. The full matrix, nine runs, one model:

ctx	f16 (loader)	q8_0 (loader)	q4_0 (loader)	q8_0 ÷ f16	q4_0 ÷ f16
4,096	576.00 MiB	306.00 MiB	162.00 MiB	0.53125	0.28125
16,384	2304.00 MiB	1224.00 MiB	648.00 MiB	0.53125	0.28125
32,768	4608.00 MiB	2448.00 MiB	1296.00 MiB	0.53125	0.28125

Right two columns are arithmetic, not vibes: 17/32 and 9/32, exact — the block sizes make q8_0 store 8.5 bits and q4_0 4.5 bits per element, and the loader agrees to the byte on all nine rows.

The crossover, division printed: 2,381.59 MiB × 1024 ÷ 144 KiB/token = 16,935.7. Call it 17k. Below that, weights dominate; above it, you are officially paying more in rent than the flat cost.

Does KV quantization cost speed?

Claim: q8_0 is free; q4_0 costs about 9% of generation speed at depth — and the difference is bigger than the run-to-run wobble. Same nine runs, generation speed at each context:

ctx	f16 gen	q8_0 gen	q4_0 gen
4,096	7.04 tok/s	6.98 tok/s	6.68 tok/s
16,384	7.07 tok/s	6.99 tok/s	7.04 tok/s
32,768	7.19 tok/s	6.97 tok/s	6.55 tok/s

Eval-time tok/s from verbose timing, 4-token generations. The wobble yardstick: prompt eval swung 13.35–14.0 tok/s across all nine identical setups — about ±0.3. f16→q8_0 at 32k is 0.22 (inside wobble); f16→q4_0 is 0.64 (−8.9%), past it.

The honest ledger: this is a CPU bench, 4-token generations — enough to price the dequant tax at 32k, not to write a manifesto. Prompt speed never moved (13.4–14.0 everywhere); the bill arrives only when generation walks a long cache. Our earlier GPU record agrees in direction: at 64K context, f16 KV ran 116 tok/s vs q4_0's 94 — on the bench where cache type was a VRAM necessity, not a speed choice.

When should you quantize the KV cache?

Claim: quantize when memory is the wall — q8_0 first, q4_0 only when the alternative is not running at all. The decision table:

• Everything fits at f16 — leave it. Lossless by construction, and it was the fastest at depth in both our benches.
• Context doesn't fit — q8_0: half-ish the cache, speed flat within wobble on this bench.
• Still doesn't fit — q4_0: less than a third of f16, roughly −9% generation at depth, and quality we did not re-measure here (our needle validations were f16-side; q4_0 cache quality is the public record's problem, not a number we'll invent).

The two knobs (public record)

llama.cpp: -ctk q8_0 -ctv q8_0 (or q4_0 / f16) on the server or CLI. Ollama: OLLAMA_KV_CACHE_TYPE=q8_0 as an environment variable — the autocomplete block behind this piece ("ollama kv cache type", "ollama kv cache quantization") is people looking for exactly that variable. Never mix K and V types; the slow path is real.

Two of our own receipts belong together here: the thinking bill itemized what a single "hi" costs when a model reasons; this piece is the other utility — rent, billed by context length, silently, forever. If you only remember one number: the meter passes the purchase price near 17k tokens, on a model small enough to fit in a jacket pocket.

last tested 2026-11-03 · llama.cpp b12069-e6192de0a · Qwen3-4B Q4_K_M (2,497,280,256 B) · CPU-only VM, 4 vCPU / 8 GB · receipts archived