GGUF quantization levels, weighed live: Q4_K_M vs Q8_0 on size, VRAM and quality

■ space · the block press · fig 1

■ the answer first

Default Q4_K_M — ~4.85 bits per weight, ~1% perplexity cost, ~30% of the full-precision size.

Step up to Q6_K for code and math; keep Q8_0 for reference runs; below Q4 is an emergency ration for large models.

The whole decision is one line: size_GB ≈ bits_per_weight × params / 8.

4.85bits per weight — the default rung

~1%perplexity cost at Q4_K_M

30%of the full-precision size

01THE PRESS

one crate, four stamps — the weighbridge answers size, the gauge answers cost

FIG 1 · THE GGUF BLOCK PRESSkeys 1–5 — or the buttons · the knob dials params

~1% perplexity — THE defaultThe default. Best quality-per-gigabyte trade there is.

RUNG Q4_K_M · 4.85 BPW · SCALE 4.85 GB / FP16 16.0 GB · SIZE 30% · PARAMS 8B

the crate is the model: same weights every pass — only the stamp changes. Press 5 and watch the cost gauge dive

02THE WHOLE LADDER

the machine carries four rungs — the registry ships eight

rung	bpw	quality — the verdict	reach for it
Q2_K the ration	~3.4	severe on sub-13B — an emergency rationcoherence starts falling apart on small models	Nothing else fits, large models only. Test before trusting.
Q3_K_M the squeeze	~3.9	noticeable — the VRAM squeezenoticeable loss — only for tight VRAM on ≥14B	Tight VRAM, ≥14B models only.
Q4_K_S the fallback	~4.6	small losssmall loss — when Q4_K_M won't fit and it's close	The fallback when Q4_K_M won't fit and it's close.
IQ4_XS the i-quant	~4.3	≈ Q4_K_M — the codebook squeeze≈ Q4_K_M quality, smaller — if your backend runs i-quants	Q4_K_M slightly too big, backend supports i-quants. Slightly slower on some backends.
Q4_K_M the default	~4.85	~1% perplexity — THE defaultbest quality-per-gigabyte — every registry ships it	The default. Best quality-per-gigabyte trade there is.
Q5_K_M the headroom	~5.7	~0.5% perplexitynear-nil loss, still comfortable	Headroom with comfort — quality mattered more than the last gigabyte.
Q6_K the workhorse	~6.6	near-indistinguishablenear-indistinguishable — the code & math rung	Code, math, extraction — anything that punishes noise.
Q8_0 the reference	~8.5	effectively none (~0.02% vs full)effectively none (~0.02%) — A/Bs and fine-tune bases	Reference runs, A/Bs, fine-tune bases — not a daily driver.

the one line—size_GB ≈ bits_per_weight × params / 8 — every readout on the press computes it live

worked example—8B at Q4_K_M: 4.85 × 8 / 8 = 4.85 GB; FP16 would be 16.0 GB — the crate leaves at 30% of full precision

weights only—add the KV cache, activations, buffers: a "4.9 GB" model wants a 6 GB card at 4k context — flash-attention + KV-cache quant keep a 16k window on that card. At a full 128k window the cache alone wants ~16 GiB.

on the hub—prefer a single Q4_K_M.gguf over sharded splits unless only the latter ships. Headed for vLLM instead of llama.cpp? Its GGUF plugin takes the same files — GPU-only rule applies. Either way the engine is a separate axis from the quant.

03WHAT THE STAMP REMOVES

the crate shrinks — here is what the stamp actually does to the bits

trained in 16-bit—every one of a model's billions of weights is a 2-byte number in 16-bit floating point; quantization packs each weight into fewer bits

the naive way—round everything to a 4-bit integer — destroys small-but-important values

block scaling—weights grouped in blocks of 32, each block carrying its own scale factor, so a wide range of magnitudes survives

k-quants, the K—importance-aware super-blocks of scales — attention and feed-forward layers treated differently, because they tolerate compression unequally

the IQ family—codebooks borrowed from image compression — fewer bits at similar quality, slightly slower on some backends

untouched—architecture, tokenizer, context handling: quantization changes only the stored weights

A Q4 file and a Q8 file of the same model are the same model, wearing different coats.

04THE DECISION ORDER

the mechanism is honest — so which rung holds which job

Context + KV budget first, weights second. Context you can't fit is worse than quality you can't measure — at long context the cache can outweigh the model.
Default to Q4_K_M. ~1% perplexity for ~30% of the size; the community default for a reason.
Step up to Q6_K when the task punishes noise — code, math, extraction, anything unattended.
Q8_0 for reference only — A/B tests, damage measurement, fine-tune bases. As a daily driver it buys warm VRAM sensors.
Below Q4 only under duress, large models only — and test with a known-hard prompt before trusting it.

higher better?—technically yes; perceptually barely — Q8_0 lands within ~0.02% of full precision: lossless for any practical purpose; Q6_K is near-indistinguishable

the naive round—Q4_0 costs a bit more than Q4_K_M's roughly 1–2%

size buys headroom—a 70B survives Q3 as tailoring; a 7B at Q2 is amputation — large models are more redundant

the floor moves—chat tolerates Q4; exact code, math, and RAG over precise documents expose the noise sooner — if a Q4 model writes subtly wrong code, test the same model at Q6_K before you blame the model

05MEASURE THE DAMAGE YOURSELF

the dials read llama.cpp's verdicts — make your own hardware vote

build llama.cpp once

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build && cmake --build build --config Release -j

pull two rungs — perplexity, then speed

huggingface-cli download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
  Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf Meta-Llama-3.1-8B-Instruct-Q8_0.gguf \
  --local-dir models
# perplexity (lower = closer to the original model)
./build/bin/llama-perplexity -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99
./build/bin/llama-perplexity -m models/Meta-Llama-3.1-8B-Instruct-Q8_0.gguf  -ngl 99
# and speed on the same hardware
./build/bin/llama-bench -m models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99

verify: the Q4_K_M number lands a few hundredths of a point worse — and the file is ~40% smaller. Same data, every rung

Perplexity is next-token surprise on one fixed dataset: a proxy, not a grade — and your real task can disagree with it.

the older mistake—agonizing over Q4-vs-Q5 while ignoring context length: budget the KV cache first; the rung is the easy part

privacy—quantization is arithmetic on weights, entirely offline: a Q4_K_M run locally leaks exactly as much as the full-precision model — nothing leaves the machine. The privacy variable is where inference runs, not the bit width

provenance—cuts the same across rungs: a stolen-base "uncensored" fine-tune in Q8 is not safer than the same weights in Q4

■ honest stamp Quality verdicts from perplexity runs, not vibes. Tested on: llama.cpp llama-perplexity · llama-bench · GGUF Q2_K → Q8_0 · Hugging Face files. The ladder's numbers are llama.cpp's own measurements on Llama-family reference models; size math is the one line — every press readout above is that line. Run the commands and replace my numbers with yours.

end of the block press · nothing fetched · everything local