These ten questions are not invented — every one is a real query from this site's own search console, and for a long stretch we ranked for each with zero clicks. The answers come from the official vLLM docs and our tested Can vLLM Run GGUF? Yes — on GPU Only deep dive.
1. Can vLLM run GGUF?
Yes — through the official vllm-gguf-plugin, on NVIDIA and AMD GPUs. Both halves matter: the plugin is vLLM's own, and the hardware table stops at GPU.
2. Does vLLM support GGUF on CPU?
No. No CPU kernel sits behind the GGUF path — x86/Arm CPUs and Intel GPUs are outside the supported table. This is the trap behind the classic report: the same GGUF that fails in vLLM on a laptop runs fine in Ollama, because GGUF's home turf is CPU-first engines. The file was never the problem.
3. How do you serve a GGUF model with vLLM?
vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B
The syntax is repo:quant_type. The trap is --tokenizer: it names the original model repo, never the GGUF repo — GGUF files carry weights, and vLLM wants the real tokenizer from the source model.
4. Which GGUF quants does vLLM support?
The K-quants people actually download — Q4_K_M and friends — work; exotic schemes may not. llama.cpp stays the reference implementation, so new quant schemes land there first and reach the plugin later.
5. Is vLLM's GGUF support production-ready?
The docs answer this one themselves: "GGUF support in vLLM is highly experimental and under-optimized." That sentence sits in the first paragraph of the official GGUF page — more honesty than most experimental features get. Treat the plugin as a way to reuse files you already have, and keep the exit to llama.cpp open.
6. Does vLLM load GGUF faster than llama.cpp?
No — and speed was never the point. llama.cpp memory-maps the file, so a GGUF starts lazily; vLLM loads it like any other checkpoint. GGUF inside vLLM is a VRAM-footprint feature, not a throughput play.
7. vLLM or llama.cpp for GGUF — which wins?
Same file, opposite instincts: llama.cpp treats GGUF as the format, vLLM treats it as an option. One GPU serving several people → vLLM plus the plugin. One person, any hardware, widest quant choice → llama.cpp, one binary.
8. Why does my GGUF file fail in vLLM?
Three real causes: plain-path serve syntax instead of repo:quant_type, a missing or wrong --tokenizer, or hardware outside the supported table. Rarer fourth: a quant scheme the plugin doesn't cover.
9. Does Ollama run GGUF?
Yes — GGUF is Ollama's native format. The same file that fails in vLLM on a CPU box usually just works there.
10. Can several users share one vLLM GGUF server?
That is the actual reason to do this: continuous batching lets one GPU serve concurrent users off the same GGUF — the users queue instead of the model reloading.
The full head-to-head — CPU, concurrency, quant coverage, setup — plus the failure-mode table live on the canonical post. One post a day, receipts included: the letters.
Top comments (5)
All ten questions came from this site's own search console — every one a query the site already ranked for with zero clicks. The surprise doing the editing: the mandatory --tokenizer flag trips more people than CPU support does. Which of the two bit you?
@max_quimby agreed, and it deserves the loudness — the instinct to point vLLM at a tokenizer directory comes straight from the HF playbook, and the GGUF path wants the weights' own vocabulary instead. That one habit probably explains half the first-run failures people blame on the plugin.
@qisuancloud this is the note worth pinning — the weights fit but the context bill arrives separately, and it scales with max-model-len times how many sequences you run at once. People size the GPU from the checkpoint file alone and then meet OOM at the first real batch.
The
--tokenizerpoint deserves to be louder — it's the single thing that trips people the first time, because the obvious instinct is to point it at the GGUF repo you just downloaded. Took us a confused debugging session to internalize that GGUF files carry weights, not a usable tokenizer, so you always name the original model.Your framing of "GGUF inside vLLM is a VRAM-footprint feature, not a throughput play" is the right mental model and I wish more people led with it. The decision we actually make in practice: if it's one GPU serving many concurrent users, vLLM's paged attention + continuous batching wins and GGUF is just a way to fit a bigger model in the VRAM you have; if it's one user on mixed hardware, llama.cpp's mmap lazy-load is the better fit and you keep that exit open.
Have you benchmarked where the K-quant quality cliff shows up for serving specifically? Q4_K_M is the usual default but I've seen output quality on structured/tool-call outputs degrade faster than on freeform text at the same quant.
Worth adding for anyone doing the VRAM math on this setup: the GGUF weights are only half the budget. vLLM reserves a KV cache sized from --max-model-len and --max-num-seqs, and with high concurrency that cache can easily exceed the model weights themselves — so a 24GB card that comfortably fits Q4_K_M weights can still OOM at runtime if you leave the defaults alone. And given the docs flag this path as "highly experimental and under-optimized," I'd pair any quant choice with a quick quality eval on your actual workload before it takes production traffic.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.