How to Run GGUF Models Locally: Ollama, llama.cpp & vLLM

Downloaded a .gguf file and wondering how to actually run it? Fastest path: ollama run hf.co/<repo>:Q4_K_M — Ollama pulls the GGUF straight from Hugging Face and serves it. GGUF is the single-file model format that llama.cpp introduced and every local-LLM tool now speaks, so the same file runs in Ollama, llama.cpp, LM Studio, Jan, and (with caveats) vLLM. This guide covers each runner with copy-paste commands, how to pick the right quantisation for your VRAM, and the load errors you will actually hit.

What is a GGUF file?

GGUF (GGML Universal File) is a container format for quantised language models. One file holds the weights, the tokenizer, and the model metadata — nothing else to download, no config soup. The weights inside are quantised: compressed from 16-bit floats down to 4-bit (or lower) integers, which is why a 9B model that needs ~18 GB in full precision fits in ~5.5 GB as a Q4 file and runs on a gaming GPU or even a CPU.

Two things matter about a GGUF file’s name:

1. The base model — gemma-3-4b-it-GGUF is a fine-tuned Gemma 3 4B exported to GGUF.
2. The quant tag — Q4_K_M, Q8_0, IQ4_XS, and friends say how aggressively the weights were compressed. More on picking one below.

Can Ollama run GGUF models?

Yes — GGUF is Ollama’s native format, and since 2024 it can pull one straight off Hugging Face without you ever touching a file:

# Pull a GGUF quant directly from Hugging Face and chat with it
ollama run hf.co/bartowski/gemma-2-9b-it-GGUF:Q4_K_M

# The quant tag after the colon picks the file inside the repo
ollama run hf.co/ggml-org/gemma-3-4b-it-GGUF:Q8_0

Already downloaded a .gguf file yourself? Point a Modelfile at it:

# Modelfile — one line is enough
FROM ./gemma-2-9b-it-Q4_K_M.gguf

ollama create gemma9b -f Modelfile
ollama run gemma9b

Ollama decides GPU offload automatically and exposes an OpenAI-compatible API on port 11434, so anything that speaks that API can use the model. The trade-off is control: you do not choose how many layers go to the GPU.

How do you run a GGUF file in llama.cpp?

llama.cpp is where GGUF comes from — the format exists for it — so support is deepest and freshest. The llama-server binary gives you both a chat UI and an OpenAI-compatible endpoint:

# Download straight from Hugging Face (picks a matching GGUF for your machine)
llama-server -hf ggml-org/gemma-3-4b-it-GGUF --port 8080

# Or run a file you already have, with full GPU offload
llama-server -m ./gemma-2-9b-it-Q4_K_M.gguf -ngl 99 --port 8080

-ngl 99 pushes 99 layers onto the GPU; set it lower than your VRAM allows and the rest stays on CPU. That partial-offload dial is llama.cpp’s superpower — a 9B model runs fine on a 6 GB card with 20 of 48 layers offloaded, just slower. For one-shot prompting instead of a server, swap llama-server for llama-cli with the same -m flag.

LM Studio is the same engine behind a desktop GUI: drop a .gguf file into its models folder (or search Hugging Face in-app) and click load. For the tooling choice itself, the Ollama vs LM Studio comparison covers which one to put underneath your models.

Which GGUF quantisation should you download?

Default answer: Q4_K_M. It is the community sweet spot — within a percent or two of full-precision quality at roughly a quarter of the size. The ladder, largest to smallest:

• Q8_0 — near-lossless; use it if your VRAM eats 8.5 bits per weight without noticing.
• Q6_K / Q5_K_M — a step down in size, still excellent for 30B+ models.
• Q4_K_M — the default. For 7–14B models this is where quality-per-GB peaks.
• IQ4_XS / Q3_K_M — for squeezing a big model onto a small card; quality loss becomes noticeable.
• Q2_K and below — last resort; the model starts degrading into nonsense mid-sentence.

The rule of thumb for fitting: file size in GB plus ~1–2 GB of context overhead should fit in your VRAM. A 4.7 GB Q4_K_M of a 9B model is comfortable on an 8 GB card. Prefer a smaller model at a higher quant over a bigger model at a terrible quant — a Q8 4B usually beats a Q2 9B.

GGUF vs Safetensors: which format do you need?

Safetensors is the unquantised archive format — full-precision weights for training, fine-tuning, and tools like transformers and ComfyUI. GGUF is the quantised, runnable format for inference on your own hardware. You cannot fine-tune a GGUF, and you cannot run a safetensors file in Ollama or llama.cpp without converting it first (that is what the convert_hf_to_gguf.py script in llama.cpp is for). Rule: training or image pipelines → safetensors; local chat and serving → GGUF. If your search started as “gguf vs safetensors”, that split is the whole answer.

Which GGUF runner should you use?

Runner	Best for	Install	GPU offload	OpenAI-compatible API
Ollama	Set-and-forget service	curl one-liner	Automatic	Yes (:11434/v1)
llama.cpp	Max control, newest features	Build or package manager	Manual -ngl dial	Yes (llama-server)
LM Studio	Desktop GUI, model browsing	App download	Automatic	Yes (local server)
vLLM	Batched multi-user serving	pip install vllm	Automatic	Yes (native)

Pick Ollama if you want it running at boot and out of sight — it is what I use on my homelab to serve models to everything on the network. Pick llama.cpp when you need a feature the day it ships (new architectures land there first) or want layer-level memory control. Pick LM Studio for a GUI. Pick vLLM only when one model must serve many concurrent users — its GGUF support works but is second-class next to its native formats.

Why won’t my GGUF model load?

The four errors that cover most cases:

1. unknown model architecture — the GGUF uses an architecture your runtime predates (new MoE and vision models land constantly). Update Ollama or rebuild llama.cpp; no other fix exists.
2. Out of memory at load — the quant is too big for your VRAM plus context. Drop one rung (Q4_K_M → Q3_K_M), lower -ngl, or shrink context with -c 4096.
3. Download truncated / corrupt — GGUF load fails with a magic-number or metadata error. Re-download and compare the SHA256 shown on the Hugging Face page.
4. ollama run ./model.gguf refuses — expected: Ollama’s run takes model names, not file paths. Use the Modelfile route shown above.

One last angle worth knowing: a local GGUF endpoint pairs well with agentic coding tools — point an OpenAI-compatible client at it and completions cost only electricity. The best local LLMs for coding tested which models deserve the slot once the plumbing in this guide works.