GGUF Quants: Q4_K_M vs Q5_K_M vs Q8_0 & IQ (2026)

When in doubt, pick Q4_K_M and you won't go wrong. A 7–8B model fits in about 6 GB of VRAM, loses only ~1% on most benchmarks versus full precision (FP16), and runs in every tool — Ollama, LM Studio, and llama.cpp. If you have VRAM headroom (8–12 GB or more), stepping up to Q5_K_M or Q6_K gives noticeably better quality on multilingual text, structured output, and complex reasoning. This guide walks through how to read GGUF naming, the tradeoffs per level, how IQ (imatrix) quants differ, and how to choose by task.

For background on GGUF files and the tools that load them, see Ollama vs LM Studio; for estimating required VRAM, use the per-model requirements articles and the VRAM calculator.

Quantization lowers the numerical precision of a model's weights to shrink file size and memory use. The original model stores each weight in 16 bits (FP16/BF16); compressing to 4 or 5 bits cuts memory to roughly a third or a quarter. The cost is a small quality loss from rounding. The essence of choosing a quant is knowing where the line is — how far you can compress before it stops being usable.

A name like Q4_K_M packs three pieces of information. Break it apart and it becomes readable.

So Q4_K_M means "4-bit, K-quant method, Medium size." The Q4_0 / Q4_1 names without _K are legacy methods; today, choosing a K-quant (_K_M / _K_S) is almost always enough. The K_M vs K_S difference: K_M spends more bits on the quality-critical attention layers for a tiny size increase, so K_M is almost always worth it.

Using an 8B-class model as the example, here are rough figures per level (numbers vary by model size).

The practical rule is simple: Q4_K_M for a 6–8 GB GPU, Q5_K_M–Q6_K for 8–12 GB, Q8_0 if you have room to spare. Below 3-bit is a compromise for "I just need it to run" — avoid it for daily use.

Names starting with IQ (e.g. IQ4_XS, IQ3_M) use an importance matrix (imatrix) during quantization. Where ordinary quantization treats all weights equally, imatrix measures which weights actually drive the output using real data, and keeps the important ones at higher precision.

- Upside: higher quality at the same bit count. IQ4_XS is often a drop-in replacement for Q4_K_M, saving ~250 MB on an 8B model
- Caveat 1 (speed): I-quants are heavier to decode and can be slower than K-quants on CPU or Apple Silicon. On NVIDIA GPUs the gap is small
- Caveat 2 (quality): if you quantize yourself, imatrix calibration is essential; I-quants without imatrix can degrade noticeably. If you use a pre-built GGUF (imatrix already applied), you don't need to worry about this

In short: IQ quants if you want it smaller and high-quality on an NVIDIA GPU; K-quants (Q4_K_M etc.) if you prioritize speed on CPU/Mac.

Quantization doesn't hurt every task equally. Testing through early 2026 confirms that commonsense reasoning and general chat are resilient (barely affected even at 4-bit), while arithmetic and strict reasoning hit a quality cliff below 4-bit. For coding and long reasoning chains, the single step from Q4 to Q5 matters most. Conversely, for summarization, chat, or classification, Q4_K_M is usually plenty.

This "different tasks need different precision" property also informs model choice: whether a small model at high-precision quant beats a large model at low-precision depends on the task. See How to Choose a Local LLM (as of June 2026) for more.