Can vLLM Run GGUF? Yes — on GPU Only

Last week a reader asked why their GGUF file 404s inside vLLM on a CPU box while the same file runs fine in Ollama. Short answer, before anything else: yes, vLLM runs GGUF — through an official plugin, on GPU only. The plugin is vllm-gguf-plugin, the syntax is repo:quant_type, and the moment you try it on a CPU you are outside the supported hardware table. Everything below comes from the vLLM documentation and the vllm-project/vllm repo record (92,000+ stars since February 2023) — the docs, not my benchmark rig.

Does vLLM support GGUF? Yes — as an out-of-tree plugin, on GPU only. Both halves of that sentence matter, and each gets a section below. This post belongs to the run-LLMs-locally cluster; the CPU-side alternatives live there too.

How do you serve a GGUF model with vLLM?

The plugin is the whole trick.

• GPU present? No GPU — stop here, llama.cpp is the CPU path
• uv pip install vllm-gguf-plugin — a plain pip install vllm is not enough
• vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B
• Two GPUs? Add --tensor-parallel-size 2
• Restart the server and run the vllm serve command again to confirm — clean start, no long tokenizer conversion

Two steps: install the plugin, then point vLLM at the model. GGUF support no longer ships inside core vLLM — the docs note it "has migrated to OOT vllm-gguf-plugin", so a plain pip install vllm is not enough:

uv pip install vllm-gguf-plugin

# Directly from Hugging Face, repo_id:quant_type format:
vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M \
  --tokenizer Qwen/Qwen3-0.6B

# Or a local file you already downloaded:
vllm serve ./Qwen3-0.6B-Q4_K_M.gguf \
  --tokenizer Qwen/Qwen3-0.6B

verify: vllm serve ./Qwen3-0.6B-Q4_K_M.gguf --tokenizer Qwen/Qwen3-0.6B → serving, no slow tokenizer conversion

The --tokenizer flag is not decoration. The official docs recommend the base model's tokenizer because GGUF tokenizer conversion is "time-consuming and unstable, especially for some models with large vocabularies". Skip it and you trade a one-line flag for a long, flaky startup.

Two GPUs, one model: add --tensor-parallel-size 2 to shard the same GGUF across both cards — tensor parallelism works with GGUF the same as any other format.

Why does vLLM refuse GGUF on CPU?

The refusal is architectural.

This is the part that surprises people. GGUF's reputation is CPU-first — it is the format llama.cpp built its name on, running models on laptops and Raspberry Pis. Inside vLLM the situation inverts. The official quantization hardware compatibility table marks GGUF:

Hardware	GGUF in vLLM
NVIDIA Volta / Turing / Ampere / Ada / Hopper	supported
AMD GPU	supported
Intel GPU	not supported
x86 CPU	not supported
Arm CPU	not supported

Source: vLLM quantization docs. The reason is architectural: vLLM's GGUF path dequantizes blocks into GPU kernels built for batched serving. There is no CPU kernel behind it, because vLLM is a serving engine, not a laptop toy. If your machine has no GPU, no flag will help — use llama.cpp. And one lock sits above the hardware one: the operating system. Can you run vLLM on Windows? Not natively — WSL2 is the supported road, and skipping it costs the fork tax.

Coming from Ollama, where the same file runs on CPU?

That is the engine, not the format: llama.cpp built its name running GGUF on laptops and Raspberry Pis. The file is fine — vLLM just has no CPU kernel behind its GGUF path.

What breaks: the honest limits list

The same docs page carries a warning worth quoting verbatim: "GGUF support in vLLM is highly experimental and under-optimized." Concretely:

• Quant coverage is narrower than llama.cpp. The K-quants people actually download (Q4_K_M and friends) work; exotic schemes may not. llama.cpp remains the reference implementation for the format.
• Architecture support trails. New model families land in llama.cpp first; the plugin follows later.
• No mmap-style lazy loading. llama.cpp memory-maps the file; vLLM loads it like any other checkpoint.
• It is a memory-footprint feature first. The docs frame GGUF as a way to shrink VRAM use, not a throughput play. How much a given file actually needs — weights, KV cache, a verdict card by card — is what the GGUF VRAM calculator estimates before you download.

None of this is hidden. It is all in the first paragraph of the official GGUF page — which is more than most experimental features get.

vLLM or llama.cpp for GGUF: which wins?

Different tools that happen to read the same file:

	vLLM + GGUF	llama.cpp
CPU inference	no	yes, first-class
Concurrent users	continuous batching, built for it	limited
Quant coverage	subset, experimental	the reference
Setup	vLLM + plugin	one binary
Best for	one GPU, many users	one user, any hardware

If you serve a model to a team from a single GPU, vLLM + GGUF lets you reuse the same Q4_K_M files the rest of the local-LLM world shares — with more choices in our GGUF quantization guide. For the full engine comparison, see llama.cpp vs Ollama and how to run GGUF models locally.

The one-line rule: same file, opposite instincts — llama.cpp treats GGUF as the format, vLLM treats it as an option.