Every local-model download page is a wall of cryptic filenames: Q4_K_M, Q5_K_S, Q8_0, IQ3_XS. They all describe the same model compressed to different degrees. On an 8 GB machine, that choice decides whether the model runs at all.
This is the practical explanation — no linear algebra required.
What quantization is
A neural network is a pile of numbers (weights). Training produces them at 16-bit precision: each number uses two bytes. A 7B model has 7 billion weights, so:
- 16-bit → 7 × 2 bytes ≈ 14 GB
- 8-bit → ≈ 7 GB
- 4-bit → ≈ 3.5 GB (plus overhead, so ~4.1–4.7 GB in practice)
Quantization rounds those numbers to a coarser grid. The model keeps its structure and most of its behaviour, but each weight takes less space. It is lossy compression for math.
The reason this works is that neural networks are tolerant of small errors in individual weights. What they are not tolerant of is running out of memory mid-answer.
Reading the names
| Label | Bits per weight | 7B size (approx.) | Use it when |
|---|---|---|---|
Q8_0 |
8 | 7.2 GB | Small models (≤1B), or 16 GB+ RAM |
Q6_K |
~6.6 | 5.5 GB | Quality matters and you have headroom |
Q5_K_M |
~5.2 | 4.8 GB | 12 GB RAM, quality-sensitive work |
Q4_K_M |
~4.5 | 4.1–4.7 GB | The default for 8 GB |
Q4_0 |
4 | 3.8 GB | Older format; prefer K-quants |
Q3_K_M |
~3.5 | 3.3 GB | Last resort for very tight memory |
IQ2_XXS |
~2 | 2.0 GB | Emergency only; quality drops sharply |
The K variants use a block scheme: groups of weights share a scale, and sensitive tensors get more bits. S, M and L control how generous that allocation is. M is the usual sweet spot.
IQ (importance-matrix) quants use a calibration pass to decide which weights matter most. They are better at very low bit rates, but slower to produce and less widely available.
Why Q4_K_M is the right default at 8 GB
The quality curve is not linear. Going from 8-bit to 6-bit loses very little. 6 to 5 is small. 5 to 4 is noticeable but acceptable. Below 4, degradation accelerates — especially for small models, which have less redundancy to spare.
At the same time, memory headroom is not linear either. On this laptop:
| Quant | File | Free RAM after load | Result |
|---|---|---|---|
| Q8_0 (7B) | 7.2 GB | negative | Will not load |
| Q6_K (7B) | 5.5 GB | ~0 GB | Swaps immediately |
| Q5_K_M (7B) | 4.8 GB | ~0.7 GB | Works, fragile |
| Q4_K_M (7B) | 4.7 GB | ~0.8 GB | Works, fragile |
| Q4_K_M (3B) | 2.0 GB | ~3.5 GB | Comfortable |
The jump from Q5 to Q4 buys almost nothing on a 7B model — both are at the edge. The real win is choosing a smaller model at Q4 instead of a larger one at Q2.
The rule I use
Pick the largest model that fits at Q4_K_M with at least 1 GB to spare. Do not push a bigger model down to Q2 to make it fit; a well-quantized 3B usually beats a mangled 7B.
The memory nobody counts: the KV cache
Model size is only part of the footprint. During a conversation, the model stores attention keys and values for every token in the context. That cache grows with:
- context length — doubling the context roughly doubles the cache
- number of layers and heads — bigger models cache more
- batch size — usually 1 for local chat
For a 7B model at 8k context, the KV cache can add 1–2 GB. That is enough to turn a “barely fits” model into a swapping one halfway through a long chat.
Practical fixes: keep context at 4k, start a new chat when the topic changes, and avoid pasting huge documents into a 7B model on 8 GB.
How to choose in practice
- Start at Q4_K_M. It is the best-documented, most widely available quant.
- Check the file size against your free RAM (not total RAM). Leave 1 GB for the OS and cache.
- If it swaps, drop a model size before dropping a quant level. A 3B at Q4 is better than a 7B at Q2.
- If the model is under 1B, use Q8. The file is small enough that the extra precision is nearly free.
- For code models, prefer Q5_K_M if you have 12 GB+. Code is more sensitive to quantization noise than prose.
Downloading the right file
On Hugging Face, look for the “Files and versions” tab and pick a single .gguf file. Avoid splits (-00001-of-00003) unless your runtime supports them; Ollama and recent llama.cpp do, but it is extra friction.
With Ollama, the quant is usually baked into the tag:
ollama pull qwen3:1.7b # default quant chosen by the publisher
ollama pull llama3.2:3b
For explicit control, download a GGUF and point llama.cpp at it:
llama-cli -m ./models/qwen3-1.7b-q4_k_m.gguf -c 4096 -t 4 -p "Hello"
-c 4096 sets the context, -t 4 sets threads to physical cores. Both directly affect memory use and speed.
What quantization does not fix
- A slow CPU is still slow. Q4 makes the model smaller; it does not make matrix multiplication faster per byte.
- A bad model is still bad. Quantization preserves the model's strengths and weaknesses.
- Long context is still expensive. The KV cache is separate from the weights.
- Thermals. Smaller models run cooler, but sustained inference still throttles a thin laptop.
FAQ
Can I convert a model to GGUF myself?
Yes, with llama.cpp's conversion scripts, but it needs the original weights and a lot of disk. For most people, downloading a pre-quantized GGUF is the sane path.
Why are there two Q4 files, Q4_0 and Q4_K_M?
They use different quantization schemes. K-quants generally preserve quality better at the same size. Prefer Q4_K_M unless a specific runtime or model only offers Q4_0.
Does Ollama use GGUF?
Yes. Ollama pulls GGUF weights and wraps them with a Modelfile and runtime configuration. The underlying quantization is the same.
What about AWQ, GPTQ and EXL2?
Those are GPU-oriented formats. They are faster on supported GPUs but not the right choice for CPU inference on 8 GB. Stick with GGUF here.
Related reports
- Best small LLMs for 8 GB
- Can you run an LLM on 8 GB?
- Ollama vs llama.cpp
- Benchmark table: size, speed and headroom