How to run GGUF models locally: the pre-flight, from download to serving

■ space · the first flight

■ the answer first

Fastest path: ollama run hf.co/<repo>:Q4_K_M — Ollama pulls the GGUF straight from Hugging Face and serves it.

Want the -ngl dial (which layers hit the GPU)? llama-server -m file.gguf -ngl 99.

Quant default: Q4_K_M — file size + ~1–2 GB context overhead should fit your VRAM.

THE PRE-FLIGHT

Five checks between download and a serving model. Tick them as you go — the runway lights up behind you.

■ HOLDING — the checklist is waiting.

THE FILE, AND ITS NAME

GGUF (GGML Universal File) is a container for quantised models: weights, tokenizer, metadata — one file, no config soup.

Weights compressed from 16-bit to 4-bit: a 9B needing ~18 GB at full precision fits in ~5.5 GB as a Q4 — gaming GPU or even CPU.

bartowski/gemma-2-9b-it-GGUF:Q4_K_M

the org · the base model · the export · the quant tag — two things matter: the base (gemma-2-9b-it) and how hard the weights were squeezed (Q4_K_M)

TWO WAYS TO FLY IT

Ollama — the set-and-forget service. GGUF is native, and since 2024 it pulls straight off Hugging Face. The trade-off is control: you don't choose GPU layer counts.

# pull a GGUF quant directly from Hugging Face and chat
ollama run hf.co/bartowski/gemma-2-9b-it-GGUF:Q4_K_M

# or wrap a file you already have — Modelfile, one line:
# FROM ./gemma-2-9b-it-Q4_K_M.gguf
ollama create gemma9b -f Modelfile
ollama run gemma9b

llama.cpp — where GGUF comes from. Support is deepest and freshest, and llama-server gives you both a chat UI and an OpenAI-compatible endpoint:

# download straight from Hugging Face (picks a matching GGUF)
llama-server -hf ggml-org/gemma-3-4b-it-GGUF --port 8080

# or run a file you already have, with full GPU offload
llama-server -m ./gemma-2-9b-it-Q4_K_M.gguf -ngl 99 --port 8080

The -ngl dial is llama.cpp's superpower — partial offload. Try it on the record's example, a 48-layer 9B on a 6 GB card:

layers on the GPU (-ngl)

20 of 48 layers on the card, the rest riding CPU through PCIe — it runs fine on a 6 GB card, just slower.

One-shot prompting: swap llama-server for llama-cli, same -m flag. LM Studio is the same engine behind a GUI — drop in a .gguf, click load.

WHICH QUANT IN THE TANK

Default answer: Q4_K_M — within a percent or two of full precision at roughly a quarter of the size; quality-per-GB peaks here for 7–14B models.

The ladder, largest to smallest:

Q8_0 — near-lossless; use it if your VRAM eats 8.5 bits per weight without noticing.
Q6_K / Q5_K_M — a step down in size, still excellent for 30B+ models.
Q4_K_M — the default.
IQ4_XS / Q3_K_M — squeezing a big model onto a small card; quality loss becomes noticeable.
Q2_K and below — last resort; the model starts degrading into nonsense mid-sentence.

The fitting rule: file size + ~1–2 GB of context overhead ≤ VRAM — a 4.7 GB Q4_K_M of a 9B is comfortable on an 8 GB card.

The overhead is the KV cache: it scales with context, not model size. Budget for your longest real session.

And prefer a smaller model at a higher quant — a Q8 4B usually beats a Q2 9B.

Format corner: safetensors is the unquantised archive — training, fine-tuning, transformers, ComfyUI. GGUF is the quantised, runnable one.

You can't fine-tune a GGUF, and you can't run safetensors in Ollama or llama.cpp without converting (convert_hf_to_gguf.py is the bridge). Training → safetensors; local chat → GGUF.

THE FOUR ABORTS

Why won't my GGUF model load? These four cover most cases:

01unknown model architecture

The GGUF uses an architecture your runtime predates — new MoE and vision models land constantly. Update Ollama or rebuild llama.cpp; no other fix exists.

02Out of memory at load

The quant is too big for your VRAM plus context. Drop one rung (Q4_K_M → Q3_K_M), lower -ngl, or shrink context with -c 4096.

03Download truncated / corrupt

Load fails with a magic-number or metadata error. Re-download and compare the SHA256 shown on the Hugging Face page.

04ollama run ./model.gguf refuses

Expected: run takes model names, not file paths. Use the Modelfile route above.

Last angle: a local GGUF endpoint pairs beautifully with agentic coding tools — point an OpenAI-compatible client at it; completions cost only electricity.

Once the plumbing works, the question is which model deserves the slot; the engine-bay piece reads the llama.cpp-versus-Ollama record end to end.

One model, many users? vLLM's GGUF mode works — GPU-only, second-class next to its native formats.

■ honest stamp Tested on: ollama · llama.cpp builds · LM Studio · vLLM · GGUF Q4_K_M. First published 2026-09-07, record refreshed 2026-10-06. Every command above runs as printed; the sizes (5.5 GB Q4 / ~18 GB fp16 for a 9B), the 6 GB-card-with-20-of-48-layers example and the four aborts are the tested record of the original piece. No new benches were run to retype them.

end of the first flight · nothing fetched · everything local