mrsaynothing.dev
What happens when Ollama runs out of VRAM? The offload cliff
Ollama never refuses a model that misses VRAM — it splits it. Fitted layers stay on the GPU, the rest fall back to CPU, and tok/s falls with them. How to see the split in one line, and the three honest fixes, ranked.
When a model needs more VRAM than the card has, Ollama offloads by layer: the layers that fit stay on the GPU, the remainder runs on the CPU. Nothing crashes and nothing warns — the model just runs at CPU speed from the split point down.
ollama psprints the split as its PROCESSOR column, and the honest fixes are a smaller quant, a smaller model, or a shorter context.
What does offload actually mean?
A GGUF model is a stack of transformer layers, and a layer is atomic: it runs wholly on the GPU or wholly on the CPU. At load time Ollama measures the free VRAM, packs in as many layers as fit, and leaves the rest for the CPU. The weights themselves are only half the booking — the KV cache that attention needs grows with your context window on top. So the split you get is not a property of the model alone; it is the model, at your context length, on your card, today.
The practical consequence: offloading is a speed decision, not a correctness one. The output is the same. The time it takes to arrive is not.
How do I see the split?
One command:
$ ollama ps
NAME SIZE PROCESSOR UNTIL
d3:2b 4.6 GB 100% GPU 4 minutes from now
d3:4b 8.1 GB 38%/62% CPU/GPU 4 minutes from now100% GPU is the only line that means fully fitted. A mixed line reads CPU-share first: 38%/62% CPU/GPU is more than half the model on the CPU, no matter how fast the card is. The Ollama server log says the same thing at load time — it prints how many of the model's layers were offloaded to the GPU, which is the number to watch when you tune.
Why does a model that "fit" stop fitting?
Because the context window moved. Weights are static; the KV cache grows with every token of context the model must attend over. A model that fits at the default window stops fitting the moment you raise num_ctx to work with long documents — and Ollama will happily re-balance, silently pushing two more layers onto the CPU. If your slowdown appeared after a context bump, that bump is your answer.
The fixes, ranked
| fix | cost | when it wins |
|---|---|---|
| smaller quant — Q8_0 → Q4_K_M roughly halves the weights | some quality, usually the least-trafficked digits | you are a few hundred MB over the line |
| smaller model — one rung down the family | capability, honestly lost | the split is a third or more of the model |
shorter context — num_ctx back down | documents you can work with at once | the slowdown started with a context bump |
Ranked by what they cost you: a quant step is usually the cheapest, a model step is the honest one, and a context step is free if you never needed the window in the first place.
How bad is the cliff?
Our own bench ledger from this week — the CPU box that refused to chat — gives the floor: an 8-core, no-GPU box stamps d3:2b verdicts in 0.8–2.6 s warm, after an 11.8 s cold load, with the 3.7 GB rung resident. That is a decision model scoring tickets — short prompts, single-token answers — and it is fine. Chat is where CPU speed hurts: long prompts, long generations, every token billed at CPU rate. The cliff is not a wall; it is a slope, and how far you slide depends on what you ask.
Read-from-record piece: the receipts come from our 2026-10-24 CPU bench ledger, the Ollama library's size rungs, and ollama ps output on this box. No new bench today — the record already carried the numbers.
A model that half-fits is the worst deal on the shelf: all of the VRAM bought, none of the speed delivered. Check the split before you check the clock.