能跑，但输出不对Windows Linux macOS最后更新 2026年11月6日

Which GGUF quant should I download? (Q4 vs Q5 vs Q8)

Twenty files on the download page and I don't know which one

Which GGUF quant should I download?

你遇到的是哪一种？
如果你看到	原因是	前往
Your VRAM is at least the Q4_K_M file size plus 1–2 GB	Q4_K_M fits with room for the context cache	Fix 1: start at Q4_K_M and go up if space remains
Only a Q3 or Q2 file fits your card	The model is too big for this card at a sensible quant	Fix 1: consider a smaller model at Q4 instead
Output is fine for chat but sloppy in code or JSON	The task is sensitive to quantization	Fix 3: use Q5_K_M or Q6_K if it fits
The repo lists IQ and K files of similar size	Two quantization schemes at the same bit budget	Fix 2: the quick reference

何时出现

You found the model on Hugging Face and the repo has a dozen .gguf files: Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0, and more. They range from tiny to huge and the names mean nothing yet. You just want to know which one to click.

A Hugging Face repo with a dozen .gguf files and no recommendation on the model card.
An Ollama library page where the default tag is usually Q4_K_M and the other tags spell out their quant.
A model split into several files, where you need every part of one quant rather than one part of each.
Choosing between a bigger model at a low quant and a smaller one at a high quant of about the same size.

实际原因

Quantization is how much the model's weights are compressed. Lower numbers (Q2, Q3) are smaller and faster but lose accuracy; higher numbers (Q6, Q8) are closer to the original but bigger and need more VRAM. There's no single "best" file — the right quant is the largest one that still fits your hardware with headroom for context. So the question "which quant?" is really "how much VRAM do I have?" The arithmetic behind the file sizes is simple: size is roughly parameters × bits per weight ÷ 8. Each step up the ladder — Q4_K_M, Q5_K_M, Q6_K, Q8_0 — spends more bits on every weight: a Q8_0 file is a little over half the size of the unquantized F16 one, and a Q4_K_M file under a third of it. The methodology page publishes the exact bits-per-weight figure for each, the same figures the analyzer computes with. The _M and _S suffixes mark medium and small mixes within the same bit level, and files labelled IQ use a newer scheme that keeps more quality at very low bit levels, at some cost in speed on some hardware.

如何修复

1
Pick the largest quant that fits your VRAM最常见的修复方法
The decision rule is simple: choose the highest quant your card can hold with a little room to spare for context. For most people that lands on Q4_K_M — it's the community default sweet spot, small enough to fit common cards and close enough to full precision that you won't notice the difference in chat, coding, or summarisation. Go higher (Q5_K_M, Q6_K) only if you have spare VRAM; go lower (Q3, Q2) only if Q4 won't fit. The fastest way to land on the right file is to check which quant of your model fits your exact card.
生效了吗？The model loads fully onto the GPU at the context length you intend to use: ollama ps shows 100% GPU, or the load log shows every layer offloaded.
检查哪些模型适配你的硬件see exactly which GGUF quant of your model fits your GPU打开显存检查器 →
2
Quick reference for the common quants
Q8_0 — near-lossless, large, only if you have lots of VRAM. Q6_K — excellent quality, a good step down. Q5_K_M — high quality, modest size. Q4_K_M — the recommended default; best balance for most setups. Q3_K_M — noticeably softer, for tight memory. Q2_K — smallest, clear quality loss; last resort. The _K_M variants are the modern k-quants and are preferred over the older plain Q4_0 style at the same size.
生效了吗？You can name the file you need before downloading it, and its size is below your VRAM with room left for the context cache.
3
Match the quant to your use, not just your card
Casual chat tolerates lower quants well. Coding, structured output, and tasks needing precision benefit from Q5/Q6 if you can fit them. If two quants both fit, the bigger one is the safer pick for demanding work.
生效了吗？Run the same few prompts from your real work on both quants. If you cannot tell the answers apart, keep the smaller file.

如果以上都没用

If no quant of the model fits with room to spare, the answer is a smaller model rather than a lower quant, and the analyzer shows which models fit your card. If a quant fits on paper and still fails to load, the context length is usually the missing piece of the arithmetic.

同样影响： Linux · Apple

常见问题

What is the best GGUF quant for most people?

Q4_K_M. It is the community default because it balances size, speed, and quality — it fits common GPUs and is very close to the full model for everyday use. Move up only if you have VRAM to spare.

What is the difference between Q4_K_M and Q4_0?

Both are roughly 4-bit, but the _K_M k-quants distribute precision more intelligently across the model, giving better quality at a similar size. Prefer the _K_M variants when available.

Is a bigger model at low quant better than a smaller model at high quant?

Often yes — a larger model at Q4 usually beats a much smaller model at Q8 of similar total size. But both must fit your VRAM with room for context, which is the real constraint to check first.

How do I know which quant fits my GPU?

Compare the file size and the model’s memory needs against your VRAM, leaving headroom for the KV cache. Rather than guessing from file sizes, check your card against the model to see which quants fit.

What does K_M mean in Q4_K_M?

K marks llama.cpp's k-quant scheme, which quantizes weights in blocks with their own scales, and M means the medium mix, which keeps some sensitive tensors at a higher precision than the rest. K_S is the small mix and saves a little more space. K_M is the usual choice at each bit level.

Are IQ quants better than K quants?

At very low bit levels, often. IQ formats such as IQ3_M and IQ4_XS keep more quality per bit than K quants of the same size, which matters most below 4 bits. They can run slower on some CPUs and older backends, so at Q4 and above the K quants remain the safe default.

Which quant does Ollama download by default?

Most models in the Ollama library default to a 4-bit quant, commonly Q4_K_M, and the model's tag page lists the quantization of every tag. To get another one, pull the specific tag that names it, such as a q8_0 tag, when the library offers it for that model.

Does a lower quant make the model faster?

Usually, because generation speed is limited by how many bytes are read per token: a Q4 file is read in about half the time of a Q8 file of the same model on the same card. The exception is when a format's decoding work costs more than the memory it saves, which can happen with some IQ formats on CPUs.