GGUF quantization: which format to pick, what it costs in accuracy, and why the same 4-bit setting cost us 0.045 once and 0.004 the next time
GGUF quantization shrinks a model to run on a laptop. File sizes by format, what llama.cpp 1.0 changes, and why you must score the quantised file itself.
By overads Inc., the team building gloofy
GGUF quantization stores a language model's weights at lower precision, typically 4 to 8 bits instead of 16, in the single-file GGUF format that llama.cpp, Ollama and LM Studio run. It cuts file size and memory by roughly three to four times, which is what lets a useful model run on a laptop. The cost is accuracy, and it is not predictable from the format name: the same 4-bit setting cost one version of our model 0.045 and the next version 0.004. Pick a format by size, then score the file itself.
What changed this week: llama.cpp reached 1.0
On 13 November, llama.cpp and its tensor library ggml released version 1.0.0, about three and a half years after the project began. The Hugging Face announcement describes it as the first stable release of both libraries, with semantic versioning from now on: a major version for breaking changes to the public C API, a minor version for new features and model support, a patch for fixes. The release removes 35 deprecated functions, adds FP8 caches for the attention memory, and adds support for new architectures including Kimi-K3.1-Pro.
For marketers running local models, the practical meaning is stability. Apps built on llama.cpp should break less often when they update, and a GGUF file you quantise today has a stable tool behind it. It does not change how quantisation itself works, or what it costs.
GGUF quantization formats and file sizes
The llama-quantize documentation lists file sizes for one reference model, Meta's Llama 3.1 8B. The 16-bit original is about 15 GiB.
| Format | Bits per weight | Size for Llama 3.1 8B | Typical use |
|---|---|---|---|
| F16 | 16.0 | 14.96 GiB | Reference, not for laptops |
| Q8_0 | 8.5 | 7.95 GiB | Near-lossless, when memory allows |
| Q6_K | 6.56 | 6.14 GiB | Cautious default for judgement tasks |
| Q5_K_M | 5.70 | 5.33 GiB | Middle ground |
| Q4_K_M | 4.89 | 4.58 GiB | The common laptop default |
| Q3_K_M | 4.00 | 3.74 GiB | Tight memory; test carefully |
| Q2_K | 3.16 | 2.95 GiB | Last resort |
The "typical use" column is our reading, not the documentation's. The README notes that memory needs currently match disk size, because the model is loaded fully into memory. A 4B model at Q4_K_M, like ours, comes to about 2.5 GB.
The README also lists IQ formats, from about 2 to 4.7 bits per weight, which use an importance matrix: a file recording which weights matter most on sample text, so precision is spent where it counts. The tool accepts one with the --imatrix option. If you build one, use text like your real task, such as ad copy and briefs, rather than generic prose.
What quantisation costs: our two results
The documentation says accuracy loss is usually measured with perplexity or KL divergence, both of which compare the quantised model's word predictions with the original's. Those are useful, but they are not your task. We measure the file on our own exam instead, and the results differed by a factor of ten.
- Older model (r22): quantising to Q4_K_M cost 0.045 on our exam, and the loss was concentrated in the judgement labels rather than spread evenly.
- Current model (r25): the same Q4_K_M setting cost 0.004, from 0.792 for the unquantised adapter to 0.788 for the GGUF file, on exam v3.1.
Same base model, same size, same format. The only safe conclusion is that quantisation loss belongs to a specific model, not to a format, and that judgement-heavy outputs are where to look first. Both files are published, so the earlier numbers stay reproducible; the details are in our build log.
Three mistakes to avoid
- Quantising an already quantised file. llama-quantize can requantise only with an explicit flag, and its README warns this can severely reduce quality compared with starting from a 16- or 32-bit model. Always quantise from the full-precision export.
- Assuming every tool keeps the file small. Hugging Face's Transformers library can load GGUF files, but for most architectures it dequantises them into a full-size model on load. Only some newer architectures stay packed, and only with an extra kernel installed. If memory is the reason you chose GGUF, run it in llama.cpp, Ollama or LM Studio.
- Scoring the checkpoint instead of the file. A fine-tuned adapter can score well and its merged, quantised export worse. Our notes on LoRA fine-tuning cover merging before quantising; the rule is to measure after every conversion step.
How to choose a GGUF quantization for marketing work
- Start from memory. Find the largest format that fits in your RAM with room left for the context window and your other apps.
- Build a small test. Fifty real examples of your task, with agreed answers. Our guide to building an LLM evaluation covers the design.
- Score two formats. Run Q4_K_M and one step up, such as Q5_K_M or Q6_K, on the same test, more than once.
- Check judgement outputs separately. Outputs that need nuance are where our losses showed, so score them on their own as well as in the overall total.
- Keep the smaller file only if the difference is inside the noise. If the gap is bigger than run-to-run variation, pay for the extra memory.
Where gloofy fits, and where it does not
gloofy-1-nano ships as two Q4_K_M GGUF files of about 2.5 GB each, under Apache 2.0, with an Ollama Modelfile: the current r25 model and the older r22, kept so published numbers can be reproduced. Its model specifications list what it runs on, including laptops with 8 GB of RAM. We publish the score of the file you download, not the unquantised adapter.
The 14B release is different: it is a LoRA adapter for Qwen3-14B, loaded with peft, not a GGUF file, and we have not published a quantised version of it. Our measurements cover our own models only; your model's quantisation cost has to be measured on your task. More on training and serving models collects in the fine-tuning and building models hub.
What to do this week
- Check which GGUF format your local tools are running, and whether it was made from a full-precision model.
- If you use Transformers for GGUF files, check memory use; it may be running a full-size copy.
- Score your current format and one larger one on 50 real examples.
- Update llama.cpp-based tools to 1.0 when your apps support it, and note the version beside your scores.
FAQ
What is GGUF quantization?
GGUF quantization stores a model's weights at reduced precision, such as 4 to 8 bits per weight, in the GGUF file format used by llama.cpp, Ollama and LM Studio. It makes files and memory use three to four times smaller than 16-bit, so models run on ordinary laptops, at some cost in accuracy that varies by model.
Which GGUF quantization should I use?
Q4_K_M is the common default for laptops, at about 4.9 bits per weight. For tasks needing nuance, test Q5_K_M or Q6_K as well. Choose the largest format that fits your memory, then score it against a smaller one on your own examples, and keep the smaller only if the difference is within normal run-to-run noise.
How much accuracy does 4-bit quantization lose?
It depends on the model. Quantising two versions of our 4B model to the same Q4_K_M format cost 0.045 on one and 0.004 on the other, on our marketing exam. The larger loss was concentrated in judgement labels. Measure the exact file you will run, on your own task.
What does llama.cpp 1.0 change?
Released on 13 November 2026, llama.cpp 1.0.0 is the first stable release of llama.cpp and ggml, with semantic versioning so breaking API changes come only with major versions. It removed deprecated functions and added FP8 attention caches and new model support. It does not change how GGUF quantization affects accuracy.