128k context on a desktop is a lease — the KV cache is the bill, counted live

■ space · the machine shop · fig 1

■ the answer first

Llama 3.1 8B at 131,072 tokens in fp16: KV cache ≈ 16 GiB — as big as the weights.

The "8B that fits in 16 GiB" becomes a 32 GiB problem.

The cache is linear in context and allocated up front — RAM pays for 128k whether or not you fill it.

The fix is boring: set context to what you actually use — a 32k window costs a quarter of the 128k cache.

16.0GiB — the 128k kv cache · llama 3.1 8b · fp16

131,072tokens billed whether filled or not

128 KiBstanding charge — every token, llama 8b

01THE METER DESK

the tenant is the kv cache — the wheels spin the bill the card never prints

FIG 1 · THE KV-CACHE METER DESKkeys 1 · 2 · 3 · 4 — or the buttons

CACHE = WEIGHTSat full window the cache is as big as the model it belongs to — a 32 GiB tenancy now

TENANT LLAMA 3.1 8B · WINDOW 128k · 131,072 TOKENS · CACHE 16.00 GiB · WEIGHTS 16.0 GiB · SHARE 100% · FED 0

the wheels billed before the odometer moved — allocation is up front, filling is optional

02THE FORMULA THE CARDS OMIT

one public formula drove every digit on those wheels

the arithmetic — as laid out in transformer inference arithmetic

cache_bytes = 2 × layers × context × kv_heads × head_dim × bytes_per_element

"The context window is not a feature you toggle. It is a memory lease you pay every second the server runs."

the whole check — two lines against the public config.json files

def kv_cache_gib(layers, kv_heads, head_dim, ctx, bytes_per=2):
    return 2 * layers * ctx * kv_heads * head_dim * bytes_per / 2**30

print(kv_cache_gib(32, 8, 128, 131_072))  # Llama 3.1 8B  -> 16.0
print(kv_cache_gib(28, 4, 128, 131_072))  # Qwen2.5 7B    -> 7.0

verify: kv_cache_gib(32, 8, 128, 131_072) → 16.0 · kv_cache_gib(28, 4, 128, 131_072) → 7.0

2 tensors—keys + values, cached for every layer, every KV head, every token — nothing recomputed, everything remembered

why linear—storage per token is constant, so total storage is linear in context — 128× the cache of a 1k window, for the life of the session

why up front—the serving stack allocates the full window at load — RAM pays for 128k whether or not you fill it

03TWO TENANTS, ONE DESK

a formula is arithmetic until it prices two real model cards side by side

Model (fp16)	Layers × KV heads	KV cache @ 128k	Weights	Cache vs weights
Llama 3.1 8B	32 × 8	~16 GiB	~16 GiB	~100%
Qwen2.5 7B	28 × 4	~7 GiB	~15 GiB	~47%

press key 1 on the desk — the same 128k window re-meters at 7.0 GiB for qwen2.5 7b

the model-side discount—grouped-query attention (GQA): fewer KV heads, thinner cache — architecture, not quantization

why qwen feels friendlier—its 28 layers × 4 KV heads are why the same window costs about 7 GiB — real relief on modest machines

where the heads hide—specs rarely say KV heads — you dig through config.json to find them, which tells you how the number was meant to be read

04WHERE THE BILL BENDS

the meter is honest for full attention — four bends before you sign anything

prefill · the second bill—a 100k-token prompt is chewed whole before the first output character — on a 3060, that is minutes, not milliseconds

sliding-window models—mistral-style attention caps per layer, so the cache stops growing — the formula overestimates them, in your favor

kv quantization—llama.cpp can cache KV in q8_0, roughly halving it at some quality risk — half of 16 GiB is still 8 GiB

the serving stack—vLLM takes GGUF on GPU with the same memory bill attached — quantization shrinks the weights, never the cache math

Nobody reads 128,000 tokens. Your RAM pays for every one of them anyway.

the fix is boring—set context to what your workload actually uses — 32k covers nearly every prompt a single developer sends, at a quarter of the bill

the retrieval lane—RAG's job — filling a window is the expensive way to say "read section 4"; retrieval instead of a full window, because long context is not useless, it is priced

■ honest stamp Tested on: Llama 3.1 8B · Qwen2.5 7B · fp16 · 131,072-token window — recorded 2026-09-22, refreshed 2026-10-04. Every number on this page is the public formula run against the models' public config.json files (layers, KV heads, head_dim), as laid out in Transformer Inference Arithmetic — the desk's meter is that same formula in JavaScript; no benches to buy, one print away from your own terminal.

end of the tenant ledger · nothing fetched · everything local