mrsaynothing.dev
+ create a postlogin
EN

> GGUF and the VRAM budget▋

  1. 1The best local LLM for coding: pick by VRAM first24 GB takes the 30B MoE, 12-16 the 14B at Q4, 8 the 7B — the bracket band, the two rules, the Qwen-vs-DeepSeek split, and five commands to working.
  2. 2GGUF quantization levels, weighed live: Q4_K_M vs Q8_0 on size, VRAM and qualityEvery rung from Q2_K to Q8_0 on one console — pick a level, dial your parameter count, and the size answers from one line of arithmetic.
  3. 3GGUF VRAM calculator: weigh the model before the download startsWeights, KV cache and a per-card verdict from three inputs you can re-derive by hand — the 8B '8-GB-card model' that misses by 23%, explained and run live.
  4. 4GGUF VRAM calculator: the three numbers that decideWeights + KV cache + overhead: the three numbers that decide whether a GGUF quant fits — printed as formulas, checked against four public GGUF files today.
  5. 5The 7% GhostAn 8GB card holds 8.59 decimal GB but only ~7.3 usable GiB. Sizes measured today, KV math shown, one honest fit table — the 9B the tables love fails at 8k.
  6. 6Can vLLM run GGUF? Yes — through the plugin, on GPU onlyThe plugin, the repo:quant syntax, the tokenizer flag that is never optional — and the hardware table that turns CPUs away at the door.
  7. 7Can vLLM run GGUF? Ten questions, answered firstThe FAQ hub: plugin yes, GPU only, repo:quant_type plus the tokenizer that is never optional — every answer before the explanation.
  8. 8Can vLLM run GGUF? The GPU-border ruleCan vLLM run GGUF? Yes — on GPUs only. The honest vLLM–GGUF matrix: six green hardware rows, every CPU row red, and the border rule the guides skip.
  9. 9vLLM on Windows: three doors, one supportedThe docs say OS: Linux — WSL2 is the supported road, the native fork costs the fork tax, and Ollama was probably your real answer anyway.