Ollama prompt caching — where you edit decides the bill

tl;dr

0.2 seconds against 75.6. Ollama’s prompt cache gave us a 370× faster re-evaluation on the exact same ask — and then one word, moved to the middle of that prompt, billed us 49 seconds of it back. The cache is not a setting you switch on. It is a border checkpoint, and where you edit decides what you pay.

Ollama prompt caching works automatically: when your new prompt starts with the exact tokens of a recent one, the engine reuses their KV tensors instead of re-reading them — on our bench an exact repeat re-evaluated 1,539 tokens in 0.2 s instead of 75.6 s. Every token you change forces re-evaluation of everything after it: an edit near the end cost 4.2 s, the same edit at mid-prompt cost 49.2 s, and reordering sections cost 79.9 s — more than a cold load. Keep your boilerplate byte-identical at the head of the prompt, put the volatile parts at the tail, and never reorder.

verify: six probes, one fresh VM, seed 42 · temp 0 · ollama 0.40.16 · receipts below, log promptcache-2026-11-12.log

Fig 1 — six probes, one prompt. The dim bars are the receipt: a mid-prompt edit repaid 65% of the cold bill, and reordering out-billed the cold load itself.

What does ollama’s prompt cache actually keep?

Not your text. When ollama evaluates a prompt it computes a KV tensor for every token — the attention footprint of that token against all before it — and the runner keeps those tensors in RAM while the model stays loaded. A new prompt is matched against that store token by token, from the front. Every token that continues the old prefix exactly is free. The first token that differs is the border: everything from there to the end is re-read at full price. This is the same KV residency our KV-cache piece covered for unload timing — the cache that serves speed is the cache that holds RAM (and why keep_alive decides how long either survives — the context-window piece has the sizing side of that bill).

prefix hit — free tokens. Identical bytes from position 0 onward cost nothing to re-process. 0.2 s for 1,539 tokens, measured.

first divergence — the border. One changed token invalidates every token after it. There is no partial credit backwards; the tail always re-bills.

residency — keep_alive. The cache lives as long as the model does. Unload the model and the discount dies with it; pin keep_alive if your loop is chatty with gaps.

Why did the exact repeat cost 0.2 seconds?

The probe: a 1,539-token prompt — eight numbered sections of fixture-log facts, then a question — asked cold on a fresh VM. The engine read all 1,539 tokens at 20.4 tok/s and billed 75.6 s of prompt evaluation. The identical ask, sent again with the model still resident: 0.2 s. Same endpoint, same body, nothing toggled — ollama does this with no flag, no cache: true, no warming incantation. You have been getting the discount every time you re-ask; this post is about not handing it back.

curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:4b",
  "prompt": "<your 1,500-token prompt>",
  "keep_alive": "45m",
  "think": false,
  "options": {"seed": 42, "temperature": 0, "num_predict": 128}
}'

verify: P1 cold 75.6 s → P2 repeat 0.2 s, byte-identical probe body, model resident · promptcache-2026-11-12.log

Where you edit decides what you pay

We kept the 1,539-token body and moved exactly one word, three ways. Change a word in the last sentence: 4.2 s — the shared head survived, only the tail re-read. Append a sentence after the question: 1.4 s — near-total hit, five new tokens billed at the margin. Change one word at fact 41 of 96, mid-body: 49.2 s — 65% of the cold bill, paid to relocate one adjective. The cache has the memory of an elephant and the forgiveness of a customs officer: it remembers everything, and one wrong form voids every stamp in the booklet — for everything behind the wrong form.

probe	what changed	prompt-eval	whole ask	vs cold
P1 cold	nothing — fresh VM, first ask	75.6 s	111.8 s	baseline
P2 repeat	nothing — identical body	0.2 s	26.6 s	370× faster
P3 suffix	+1 sentence after the question	1.4 s	27.2 s	54× faster
P4 mid-edit	1 word at fact 41/96	49.2 s	75.2 s	65% of cold
P5 reorder	sections 1–2 ↔ 7–8 swapped	79.9 s	135.5 s	worse than cold
P6 near end	1 word in the last sentence	4.2 s	34.0 s	18× faster

1539-token prompt · prompt-eval is the re-read bill; “whole ask” adds the 128-token generation, which stayed ~25–29 s throughout.

Why does reordering bill more than a cold load?

Because the border is at token ~1. Swap the first two sections for the last two and the shared prefix is empty — the match dies immediately and every token re-bills. But P5 billed 79.9 s, not 75.6 s: 5.7% above the cold load it threw away the discount for. We are not going to pretend we caught the culprit — the honest note is that the generation pass also ran 2.1× slower on that probe (55.4 s against 25.5–29.4 s for every other ask, same seed, same budget). Warm-server state joins the bill in ways a cache diagram does not show; the determinism piece already met this witness with a flipped token at temp 0. What we can state as receipt: a reordered prompt is not a “same prompt, different order”. It is a brand-new bill, plus noise.

The fix list

Put byte-stable boilerplate at the head of the prompt: system rules, format contracts, the fixture data. One file, one order, one spelling — forever
Put the volatile part at the tail: the actual question, the per-request delta. Edits there bill 4.2 s, not 49 s
Never reorder across requests — same data, new order = new bill, and P5 says it can out-bill cold
Pin keep_alive across a chatty-with-gaps loop; an unloaded model takes its cache discount to the grave (10.6 s re-load on our cold probe)
Watch RAM, not just speed: cached KV lives in memory for every resident request — sizing that bill is the context-window piece’s territory
Measure with prompt_eval_count + prompt_eval_duration in the response, not wall-clock vibes — generation time hides the whole story

symptom	what is happening	the fix
every request feels like a cold load	model unloading between asks — cache dies with it	raise `keep_alive`; check `load_duration` in the reply
tiny prompt edits, huge latency	edit sits mid-prompt; tail re-bills from the divergence	move the volatile part to the prompt tail
“same prompt” bills differently per run	order or boilerplate drifted between requests	freeze the head byte-for-byte; diff the prompts, not the vibes

receipt · six probes, one fresh VM (qwen3:4b · ollama 0.40.16 · 2 vCPU · prompt 1,539 tok)

P1 cold-full-ask      prompt_eval 75,646 ms   (1,539 tok · 20.4 tok/s)  wall 111.8 s
P2 exact-repeat       prompt_eval    203 ms                             wall  26.6 s
P3 suffix-only        prompt_eval  1,402 ms   (+5 tok)                  wall  27.2 s
P4 mid-edit-kill      prompt_eval 49,176 ms   (1 word, fact 41/96)      wall  75.2 s
P5 reorder-full-miss  prompt_eval 79,878 ms   (secs 1–2 ↔ 7–8)          wall 135.5 s
P6 edit-near-end      prompt_eval  4,189 ms   (1 word, last sentence)   wall  34.0 s

One prompt. Six bills, from 0.2 to 79.9 seconds — a 370× spread with no setting changed and no flag flipped. The cache was always on. The only variable was where we dared to edit.

last tested 2026-11-12

qwen3:4b · 2,497,293,931 B · sha256 359d7dd4…74e7 · ollama 0.40.16 · one fresh Arch VM, 2 vCPU — CPU bench, no VRAM claims