Can vLLM run GGUF? Ten questions, answered first

vLLM runs GGUF through the official vllm-gguf-plugin, on NVIDIA and AMD GPUs only. The serve syntax is repo:quant_type with a mandatory --tokenizer pointing at the original model. On CPU, llama.cpp stays the answer.

One GPU serving several people → vLLM + vllm-gguf-plugin. One person, any hardware, widest quant choice → llama.cpp, one binary. No GPU or Intel GPU → llama.cpp, first-class CPU. Both engines read the same GGUF; the difference is what they optimize for — pick by hardware and head-count, not by file format.

Yes — through the official vllm-gguf-plugin, on NVIDIA and AMD GPUs. The plugin is vLLM's own, not a community fork. Both halves matter: the plugin is official, and the hardware table stops at GPU.

No. vLLM has no CPU kernel behind its GGUF path — x86/Arm CPUs and Intel GPUs sit outside the supported hardware table. This is the trap behind the classic report: the same GGUF that 404s in vLLM on a laptop runs fine in Ollama, because GGUF's home turf is CPU-first engines. The file was never the problem.

One line — the repo, the quant type, and the tokenizer:

vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B

The syntax is repo:quant_type. The trap is the --tokenizer flag: it names the original model repo, never the GGUF repo — GGUF files carry weights, and vLLM wants the real tokenizer from the source model.

The K-quants people actually download — Q4_K_M and friends — work; exotic schemes may not. llama.cpp remains the reference implementation for the format, so new quant schemes and new model architectures land there first and reach the plugin later.

The docs answer this one themselves: "GGUF support in vLLM is highly experimental and under-optimized." That sentence sits in the first paragraph of the official GGUF page — more honesty than most experimental features get. Treat the plugin as a way to reuse files you already have, and keep the exit to llama.cpp open.

No — and speed was never the point. llama.cpp memory-maps the file, so a GGUF starts lazily; vLLM loads it like any other checkpoint. GGUF inside vLLM is a memory-footprint feature: the docs frame it as a way to shrink VRAM use, not a throughput play. The throughput win is batching — elsewhere entirely.

Three real causes: wrong serve syntax (plain path instead of repo:quant_type), a missing or wrong --tokenizer, or hardware outside the supported table. A fourth shows up rarer: a quant scheme the plugin doesn't cover. Work the table below top to bottom — the first three rows cover most reports.

Yes — natively; GGUF is Ollama's default format, not an add-on. Pull a GGUF from Hugging Face, point Ollama at it, done. If the GPU sits idle while it runs, that's a different problem with its own checklist.

That's the one job vLLM exists for: continuous batching queues concurrent requests against one loaded model. The GGUF plugin inherits that machinery — the honest caveat is the docs' own "under-optimized" line, so if you're sizing a shared server, measure your quant on your card before promising anyone a number. And one more trap: a GGUF that serves fine for one user can still blow past VRAM when batching multiplies the KV cache per concurrent request — the weights shrank; the cache grows with each user. Drop concurrency, shorten context, or step the quant down.