Can vLLM run GGUF? Yes — through the plugin, on GPU only

■ space · the interpreter's desk

■ the answer first

Yes — vLLM runs GGUF, via vllm-gguf-plugin, on GPU only.

Syntax repo:quant_type + --tokenizer base model; x86/Arm CPU and Intel GPU: not supported.

One GPU, many concurrent users → vLLM; no GPU or widest quant coverage → llama.cpp.

The docs' own words: "highly experimental and under-optimized" — a memory-footprint feature, not a throughput play.

AT THE DESK

Say what the file runs on. The desk checks its guest list — vLLM's official quantization hardware table — and stamps a verdict.

SUPPORTED

GPU kernels are behind the GGUF path — the plugin dequantizes blocks straight into batched serving. Proceed to the counter.

route: uv pip install vllm-gguf-plugin → vllm serve repo:quant

guest list source: the official vLLM quantization hardware-compatibility table — the refusal is architectural: no CPU kernel sits behind the GGUF path, because vLLM is a serving engine, not a laptop toy

THE COUNTER

Two steps, and a flag that is never decoration:

uv pip install vllm-gguf-plugin

# directly from Hugging Face, repo_id:quant_type format:
vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M \
  --tokenizer Qwen/Qwen3-0.6B

# or a local file you already downloaded:
vllm serve ./Qwen3-0.6B-Q4_K_M.gguf \
  --tokenizer Qwen/Qwen3-0.6B

clean start. The docs recommend the base model's tokenizer because GGUF tokenizer conversion is "time-consuming and unstable, especially for some models with large vocabularies". One line, no long startup.

GGUF support no longer ships in core vLLM — the docs note it "has migrated to OOT vllm-gguf-plugin"; a plain pip install vllm isn't enough.

Two GPUs, one model: --tensor-parallel-size 2 shards the same GGUF across both cards.

Coming from Ollama on CPU? That's the engine, not the format — llama.cpp runs GGUF on Raspberry Pis. vLLM has no CPU kernel behind GGUF.

One lock sits above the hardware one: the OS. vLLM on Windows means WSL2 — skipping it costs the fork tax.

THE HONEST LIMITS LIST

The same docs page carries a warning worth quoting verbatim: "GGUF support in vLLM is highly experimental and under-optimized." Concretely:

Quant coverage is narrower than llama.cpp. The K-quants people actually download (Q4_K_M and friends) work; exotic schemes may not. llama.cpp remains the reference implementation for the format.
Architecture support trails. New model families land in llama.cpp first; the plugin follows later.
No mmap-style lazy loading. llama.cpp memory-maps the file; vLLM loads it like any other checkpoint.
It is a memory-footprint feature first. The docs frame GGUF as a way to shrink VRAM use, not a throughput play.

None of this is hidden — it is all in the first paragraph of the official GGUF page, which is more than most experimental features get.

SAME FILE, OPPOSITE INSTINCTS

	vLLM + GGUF	llama.cpp
CPU inference	no	yes, first-class
Concurrent users	continuous batching, built for it	limited
Quant coverage	subset, experimental	the reference
Setup	vLLM + plugin	one binary
Best for	one GPU, many users	one user, any hardware

Serving a team from one GPU? vLLM + GGUF reuses the same Q4_K_M files as the rest of the local-LLM world.

One user, any hardware? llama.cpp is one binary and the whole format.

The one-line rule: same file, opposite instincts — llama.cpp treats GGUF as the format, vLLM treats it as an option.

■ honest stamp The docs, not my benchmark rig. Tested on: the public record — vLLM + vllm-gguf-plugin · NVIDIA Volta → Hopper · AMD GPU. Recorded 2026-09-23, refreshed 2026-10-12. Everything here comes from the vLLM documentation and the vllm-project/vllm repo record (92,000+ stars since February 2023): the guest list is their hardware-compatibility table verbatim, the warnings are their words in quotes, and no tokens-per-second were invented to fill the gaps.

end of the interpreter's desk · nothing fetched · everything local