GGUF quantization explained

What a GGUF file is

GGUF is a binary file format for language models, defined by the ggml project and used by llama.cpp. The specification describes it as “a file format for storing models for inference with GGML and executors based on GGML”, designed for fast loading and for “single-file deployment”. It replaced the older GGML, GGMF and GGJT formats.

The difference from a safetensors checkpoint is the metadata. Hugging Face’s documentation notes that GGUF “encodes both the tensors and a standardized set of metadata”. The metadata is a key-value store with the architecture, the tokenizer and usually the chat template. llama.cpp’s server reads the template from there by default.

Files follow a naming convention. Two parts of it matter in practice:

• A mmproj- prefix marks a sidecar file with a vision or audio encoder, loaded alongside the main model.
• A suffix like -00001-of-00002 marks one shard of a model split into several files.

How to read a quantization name

A quantization name packs three facts into a few characters: roughly how many bits each weight gets, how the weights are grouped, and whether the file mixes types.

Uploaders add their own labels too. In bartowski’s repos, Q4_K_L means the standard method “with the embeddings and output weights quantized to Q8_0”, per the model card. It is not a preset in llama.cpp’s own list, which does include Q3_K_L.

Bits per weight: the type vs the whole file

Each quantization type has a fixed cost per weight, including the scales stored next to the quantized values. A whole file usually averages more than that, because some tensors get a higher-precision type. The table compares the two using llama.cpp’s own figures.

The middle column comes from the Hugging Face GGUF docs. The right-hand columns are measurements on Meta’s Llama 3.1 8B published in the llama.cpp quantize README. The half bit between “4” and Q4_K’s 4.5 pays for the scales and minimums. The measured Q4_K_M file adds another 0.39 bits, which suggests how much the higher-precision tensors in the mixture cost.

You can switch the mixtures off. llama-quantize --pure “quantizes all tensors to the same type”.

The size arithmetic, worked for a 27B model

The estimate is one line:

file size in bytes ≈ parameters × bits per weight ÷ 8

Take Qwen3.8-27B, the base of the refusal-reduced Qwen3.8 27B variants on this site. Hugging Face counts about 27.8 billion parameters in the original checkpoint, which accepts images and text. The text-model GGUF files in huihui-ai’s GGUF repo report 27,320,697,856 parameters, with the vision encoder in a separate 0.93 GB mmproj file. The difference is consistent with that file’s size at 16 bits per weight.

At Q4_K's 4.5 bpw:          27,320,697,856 × 4.5    ÷ 8 ≈ 15.37 GB
At the measured 4.8944 bpw: 27,320,697,856 × 4.8944 ÷ 8 ≈ 16.71 GB

The repo’s file labelled Q4_K (llama.cpp treats Q4_K as an alias for Q4_K_M) is 16,810,714,400 bytes, or 16.81 GB. Working backwards gives 4.92 bits per weight, close to the estimate.

Real files rarely match the formula exactly, for three reasons:

• Different tensors get different types, and the mixture depends on the architecture and on choices made by the person quantizing.
• Embedding and output tensors can be given their own types; llama-quantize has an option for each, and bartowski’s _L files use them.
• Metadata such as the tokenizer adds a small amount.

In the examples on this page, the estimate lands within a few percent of the real file. For the memory you need to run the file, which is more than the file size, use the VRAM estimator.

Worked example: a real file list

The Llama 3.3 70B abliterated GGUF repo quantizes huihui-ai’s abliterated Llama 3.3 70B. Byte counts below come from the Hugging Face API on 14 November 2026. Effective bits per weight is bytes × 8 ÷ 70,553,706,560, the parameter count Hugging Face reports for these files.

Three things stand out. The effective rates stay within 0.16 bits of llama.cpp’s 8B measurements. Several of the larger files are split into two shards, so add the parts before comparing sizes. And the card says every file was made with llama.cpp release b4381 using an imatrix built from a linked dataset; the 24.9 MB .imatrix file sits in the repo next to the quants.

The model page for Llama 3.3 70B abliterated covers the variant itself.

GB or GiB? Check the bytes

The HauhauCS GLM-4.7-Flash card lists FP16 at 56 GB, Q8_0 at 30 GB, Q6_K at 23 GB and Q4_K_M at 17 GB. The byte counts from the Hugging Face API tell a slightly different story:

All four card figures match the GiB values, rounded. One GiB is 1.074 GB, so the gap is about 7 percent, which matters when the fit is tight. The bartowski card above uses decimal GB. When the margin is small, read the byte count on the file page.

Against the 29.94 billion parameters Hugging Face reports for these GGUF files, the effective rates are 16.01, 8.51, 6.58 and 4.85 bits per weight. Hugging Face counts 31.2 billion for the original GLM-4.7-Flash checkpoint, and the card does not say what accounts for the difference. Our GLM-4.7-Flash variant page covers the model itself.

Quality and storage tradeoffs

The quantize README is direct about the cost: quantization “may introduce some accuracy loss which is usually measured in Perplexity (ppl) and/or Kullback–Leibler Divergence (kld)”, and this “can be minimized by using a suitable imatrix file”.

The only perplexity figures we cite are the help strings in llama.cpp’s quantize.cpp, quoted exactly:

Read them narrowly. They describe one 8B model. The strings do not name the evaluation text, the baseline perplexity or whether an imatrix was used. Perplexity measures next-token prediction on text, not accuracy on your task. What carries over is the shape: for this model, the penalty grows slowly from Q8_0 to Q5_K_M and much faster below 4 bits.

We have not measured quality loss for any model on this site, and we have not tested whether quantization changes how often an abliterated model refuses. Two practical rules do have a source. First, the quantize README warns that --allow-requantize “can severely reduce quality compared to quantizing from 16bit or 32bit”, so prefer files made from a 16-bit original. Second, the bartowski card suggests a file “1-2GB smaller than your GPU’s total VRAM” when you want everything on the GPU. That is the uploader’s rule of thumb, and long contexts need more headroom.

What an imatrix does

An importance matrix (imatrix) is computed by running the model over a file of calibration text. The imatrix README describes the tool as computing “an importance matrix for a model and given text dataset”, to be used “during quantization to enhance the quality of the quantized models”. You pass the result to llama-quantize --imatrix.

I-quants rely on it by design. K-quants and legacy quants can use it too. The calibration text is the quantizer’s choice, so a card that names its dataset, as bartowski’s does, tells you more than one that does not.

Running a GGUF file

llama.cpp’s llama-server can fetch one quant directly from a Hugging Face repo with -hf (it defaults to Q4_K_M) or load a local file with -m. By default it listens on 127.0.0.1:9931, per the server README.

To fetch one quant from Hugging Face and serve it:

llama-server -hf bartowski/Llama-3.3-70B-Instruct-abliterated-GGUF:Q4_K_M -c 8192

To load a file you already downloaded, with all layers on the GPU:

llama-server -m ./GLM-4.7-Flash-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf -c 16384 -ngl all

Either way, the server answers on an OpenAI-compatible endpoint:

curl http://127.0.0.1:9931/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Summarize this changelog in three bullet points: ..."}]}'

-c sets the context size, and the KV cache for that context needs memory on top of the file. Check the card for runtime notes: the HauhauCS GLM card, for example, says the model is “Not recommended for Ollama due to chat template issues”. For choosing between running locally and calling an API, see local vs hosted LLMs. Models that run locally lists variants with GGUF files.

Glossary