mrsaynothing.dev
GGUF Quantization Levels: Q4_K_M vs Q8_0, Size and VRAM
GGUF quantization levels compared: Q4_K_M vs Q8_0 on size, VRAM and quality, plus the one rule — default Q4_K_M, step up only for code and math.
Pick Q4_K_M by default; go Q6_K or Q8_0 when you have VRAM to spare and need the last few percent of quality. GGUF quantization shrinks a model's weights from 16 bits to fewer — Q4_K_M stores roughly 4.85 bits per weight, so a 7B model drops from ~14 GB to ~4.1 GB with perplexity typically less than 1% worse than the original. The question "which gguf quantization to use" has a boringly stable answer that most of the drama online obscures. Below: the full level table — Q2_K through Q8_0 on size, VRAM and measured quality loss — the size math you can run yourself, and one command to measure the damage on your own hardware instead of trusting a stranger's benchmark. If the quant is headed for vLLM rather than llama.cpp, can vLLM run GGUF has the plugin and the GPU-only rule.
Which GGUF quantization should you use?
What does GGUF quantization actually do?
A model is trained in 16-bit floating point (FP16 or BF16): every one of its billions of weights is a 2-byte number. Quantization compresses each weight into fewer bits. The naive way — round every weight to a 4-bit integer — destroys small-but-important values, so GGUF uses two tricks:
- Block scaling. Weights are grouped into blocks (usually 32), and each block gets its own scale factor. The 4-bit values are offsets within that block, so a wide range of magnitudes survives.
- Importance-aware k-quants. The "K" in Q4_K_M means super-blocks of scales, plus treating attention and feed-forward layers differently from each other, because they tolerate compression unequally.
The "I" family (IQ4_XS and friends) goes further with information-theoretic codebooks borrowed from image compression. Same idea, fancier encoding: fewer bits per weight at similar quality, at the cost of slightly slower inference on some backends.
One clarification that prevents most confusion: quantization changes only the stored weights. Architecture, tokenizer, and context handling are untouched. A Q4 file and a Q8 file of the same model are the same model, wearing different coats.
Q4 vs Q8: is higher quantization better?
Yes, technically; no, perceptually. Using llama.cpp's own perplexity runs on Llama models as the reference: Q8_0 lands within ~0.02% of FP16 — for any practical purpose, lossless. Q6_K is near-indistinguishable. Q4_K_M gains roughly 1–2% perplexity, Q4_0 a bit more, and Q2_K is where coherent answers start falling apart on small models.
Two rules the numbers imply:
- Model size buys quantization headroom. A 70B model survives Q2/Q3 far better than a 7B model does, because larger models are more redundant. Quantizing a 7B to Q2 is amputation; quantizing a 70B to Q3 is tailoring.
- The quality floor moves with the task. Chat tolerates Q4. Exact code generation, math, and RAG over precise documents expose quantization noise sooner. If a Q4 model keeps writing subtly wrong code, test the same model at Q6_K before you blame the model.
The "use it when" column is the verdict; the rest is arithmetic.
| Level | Bits/weight | Size vs FP16 | Quality loss | Use it when |
|---|---|---|---|---|
| Q2_K | ~3.4 | ~21% | Severe on sub-13B | Nothing else fits, large models only |
| Q3_K_M | ~3.9 | ~25% | Noticeable | Tight VRAM, ≥14B models |
| Q4_K_S | ~4.6 | ~29% | Small | Q4_K_M won't fit and it's close |
| Q4_K_M | ~4.85 | ~30% | ~1% perplexity | The default. Best quality/size trade |
| Q5_K_M | ~5.7 | ~35% | ~0.5% | VRAM available, quality-critical tasks |
| Q6_K | ~6.6 | ~41% | Near-nil | Code/math, still fits comfortably |
| Q8_0 | ~8.5 | ~53% | Effectively none | Reference runs, fine-tune bases |
| IQ4_XS | ~4.3 | ~27% | ≈Q4_K_M | Q4_K_M slightly too big, backend supports i-quants |
How much VRAM does each level need?
Do the size math yourself instead of memorizing tables — it is one line:
size_GB ≈ (bits_per_weight × params) / 8
# 8B model @ Q4_K_M: 4.85 × 8 / 8 ≈ 4.9 GB
# 8B model @ Q8_0: 8.50 × 8 / 8 ≈ 8.5 GB
verify: an 8B at Q4_K_M ≈ 4.9 GB · at Q8_0 ≈ 8.5 GB — before cache and buffers
Then add the parts the formula leaves out: the KV cache (grows with context length — a few hundred MB to multiple GB), activations, and compute buffers. Practical margin: a "4.9 GB" model wants a 6 GB card at 4k context, and flash-attention plus KV-cache quantization to stay there at 16k. Weights are the headline, not the whole bill — at long context the cache can outweigh the model: 16 GiB for an 8B at a full 128k window.
Decision order, no exceptions worth memorizing:
- Compute your context + KV budget first, then weights. Context you can't fit is worse than quality you can't measure.
- Default to Q4_K_M. It is the community's default for a reason — roughly 1% perplexity for 70% of the size. Every registry, including Ollama's, ships it as the baseline.
- Step up to Q6_K when the task punishes noise: code, math, extraction, anything you feed to a pipeline unattended.
- Use Q8_0 only for reference — A/B tests, quantization-damage measurement, or a fine-tune base. As a daily driver it mostly buys warmth in your VRAM sensors.
- Go below Q4 only under duress, and only on large models. Test with a known-hard prompt before trusting it.
If you are choosing which file to download on Hugging Face, prefer a single Q4_K_M.gguf over sharded splits unless the uploader only ships the latter — fewer moving parts. And if you are choosing where to run it, the engine choice is separate: see llama.cpp vs Ollama for that axis.
How do you measure quantization damage yourself?
Benchmarks differ; your prompt is constant. Build llama.cpp once, download two levels of the same model, and measure both perplexity (lower is better) and tokens/second:
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build --config Release -j
huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf Meta-Llama-3.1-8B-Instruct-Q8_0.gguf \
--local-dir models
# perplexity on a wiki-text chunk (lower = closer to the original model)
./build/bin/llama-perplexity -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99
./build/bin/llama-perplexity -m models/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf -ngl 99
# and speed on the same hardware
./build/bin/llama-bench -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99
The Q4_K_M number will be a few hundredths of a point worse than Q8_0 and the file will be ~40% smaller. If your downstream task can't tell the difference — and for most, it can't — you have your answer without reading anyone's leaderboard.
verify: Q4_K_M perplexity within a few hundredths of Q8_0 · file ~40% smaller
Reading perplexity without fooling yourself
Perplexity is next-token surprise on one fixed dataset — a proxy, not a grade. It only means something when every quant is measured on the same data, and your real task can disagree with it. A couple of points worse on the benchmark and identical on your workload is a pass, not a loss.
Does quantization hurt privacy or local-only claims?
No — it is arithmetic on weights, entirely offline, and the quantized file is just a smaller container of the same parameters. Running a Q4_K_M locally leaks exactly as much (or little) as running the full-precision model locally: nothing leaves the machine. The privacy-relevant variable is where inference runs, not the bit width. The usual caveats about model provenance apply equally to every quant level: a stolen-base "uncensored" fine-tune in Q8 is not safer than the same weights in Q4. For the file-format side of this, see how to run GGUF models locally.
Which level should you pick?
Q4_K_M, and stop reading forums about it. Upgrade to Q6_K for precision-hungry work if VRAM allows, keep one Q8_0 around for A/B comparisons, and treat anything under Q4 as an emergency ration for large models only. The one mistake worth avoiding is symmetric: worrying about Q4-vs-Q5 while ignoring context length, which wrecks more local setups than any quant ever did. The rest of the cluster — runners, KV budgets, coding picks — lives on the local-LLM hub.