[{"data":1,"prerenderedAt":121},["ShallowReactive",2],{"learn-gguf-quantization":3},{"collection":4,"slug":5,"path":6,"title":7,"seoTitle":8,"description":9,"summary":10,"primaryKeyword":11,"cluster":12,"published":13,"updated":13,"reviewed":13,"readingMinutes":14,"html":15,"toc":16,"wordCount":48,"sources":49,"faq":86,"related":99,"tested":116,"internalLinks":117},"learn","gguf-quantization","\u002Flearn\u002Fgguf-quantization\u002F","GGUF quantization explained","GGUF Quantization Explained: Q4_K_M, Q8_0 and File Size","What GGUF is, what Q8_0, Q6_K, Q5_K_M, Q4_K_M and IQ4_XS mean, and how to estimate file size from bits per weight, checked against real Hugging Face files.","GGUF is the single-file model format of the ggml project, used by llama.cpp. A quantization name such as Q4_K_M says how many bits each weight gets and how they are grouped. File size is roughly parameters × bits per weight ÷ 8, so a 27B model at Q4_K_M, which averages a little under 5 bits per weight, comes to about 16.5 to 17 GB. Real files differ because tensors are quantized differently.","gguf quantization explained","deployment","2026-11-14",9,"\u003Ch2 id=\"what-a-gguf-file-is\">What a GGUF file is\u003C\u002Fh2>\n\u003Cp>GGUF is a binary file format for language models, defined by the ggml project and used by llama.cpp. The \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fggml\u002Fblob\u002Fmaster\u002Fdocs\u002Fgguf.md\" rel=\"noopener\" class=\"ext\">specification\u003C\u002Fa> describes it as “a file format for storing models for inference with GGML and executors based on GGML”, designed for fast loading and for “single-file deployment”. It replaced the older GGML, GGMF and GGJT formats.\u003C\u002Fp>\n\u003Cp>The difference from a \u003Ccode>safetensors\u003C\u002Fcode> checkpoint is the metadata. \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Fhub\u002Fgguf\" rel=\"noopener\" class=\"ext\">Hugging Face’s documentation\u003C\u002Fa> notes that GGUF “encodes both the tensors and a standardized set of metadata”. The metadata is a key-value store with the architecture, the tokenizer and usually the chat template. llama.cpp’s server reads the template from there by default.\u003C\u002Fp>\n\u003Cp>Files follow a naming convention. Two parts of it matter in practice:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>A \u003Ccode>mmproj-\u003C\u002Fcode> prefix marks a sidecar file with a vision or audio encoder, loaded alongside the main model.\u003C\u002Fli>\n\u003Cli>A suffix like \u003Ccode>-00001-of-00002\u003C\u002Fcode> marks one shard of a model split into several files.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Ch2 id=\"how-to-read-a-quantization-name\">How to read a quantization name\u003C\u002Fh2>\n\u003Cp>A quantization name packs three facts into a few characters: roughly how many bits each weight gets, how the weights are grouped, and whether the file mixes types.\u003C\u002Fp>\n\u003Cdiv class=\"table-wrap\" role=\"region\" aria-label=\"Scrollable table\" tabindex=\"0\">\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Part\u003C\u002Fth>\n\u003Cth>Meaning\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>\u003Ccode>Q\u003C\u002Fcode> + digit\u003C\u002Ftd>\n\u003Ctd>Quantized, with that nominal bit width: Q8, Q6, Q5, Q4 and so on.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>_0\u003C\u002Fcode>, \u003Ccode>_1\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Older round-to-nearest types with blocks of 32 weights. \u003Ccode>_0\u003C\u002Fcode> stores a scale per block; \u003Ccode>_1\u003C\u002Fcode> also stores a minimum. Hugging Face lists them as legacy types, though Q8_0 is still offered in all three repos used on this page.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>_K\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>A k-quant. Weights are grouped into super-blocks of smaller blocks, and the block scales are themselves quantized.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>_S\u003C\u002Fcode>, \u003Ccode>_M\u003C\u002Fcode>, \u003Ccode>_L\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>A mixture recipe. The k-quants pull request describes Q4_K_S as Q4_K for all tensors and Q4_K_M as using Q6_K “for half of the \u003Ccode>attention.wv\u003C\u002Fcode> and \u003Ccode>feed_forward.w2\u003C\u002Fcode> tensors”.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>IQ\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>An i-quant. Super-blocks of 256 weights, decoded with the help of an importance matrix.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>_XS\u003C\u002Fcode>, \u003Ccode>_NL\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Variants within the i-quant family. llama.cpp lists IQ4_XS and IQ4_NL as “non-linear quantization”.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>F16\u003C\u002Fcode>, \u003Ccode>BF16\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Not quantized: 16-bit floating point.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003C\u002Fdiv>\n\u003Cp>Uploaders add their own labels too. In bartowski’s repos, \u003Ccode>Q4_K_L\u003C\u002Fcode> means the standard method “with the embeddings and output weights quantized to Q8_0”, per the \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fbartowski\u002FLlama-3.3-70B-Instruct-abliterated-GGUF\" rel=\"noopener\" class=\"ext\">model card\u003C\u002Fa>. It is not a preset in llama.cpp’s own list, which does include Q3_K_L.\u003C\u002Fp>\n\u003Ch2 id=\"bits-per-weight-the-type-vs-the-whole-file\">Bits per weight: the type vs the whole file\u003C\u002Fh2>\n\u003Cp>Each quantization type has a fixed cost per weight, including the scales stored next to the quantized values. A whole file usually averages more than that, because some tensors get a higher-precision type. The table compares the two using llama.cpp’s own figures.\u003C\u002Fp>\n\u003Cdiv class=\"table-wrap\" role=\"region\" aria-label=\"Scrollable table\" tabindex=\"0\">\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Type\u003C\u002Fth>\n\u003Cth>Structure, per Hugging Face docs\u003C\u002Fth>\n\u003Cth>Bits per weight of the type\u003C\u002Fth>\n\u003Cth>Whole-file bits per weight, Llama 3.1 8B\u003C\u002Fth>\n\u003Cth>That file’s size\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Q8_0\u003C\u002Ftd>\n\u003Ctd>Blocks of 32 weights, one scale each\u003C\u002Ftd>\n\u003Ctd>not stated\u003C\u002Ftd>\n\u003Ctd>8.5008\u003C\u002Ftd>\n\u003Ctd>7.95 GiB\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q6_K\u003C\u002Ftd>\n\u003Ctd>16 blocks of 16 weights, 8-bit scales\u003C\u002Ftd>\n\u003Ctd>6.5625\u003C\u002Ftd>\n\u003Ctd>6.5633\u003C\u002Ftd>\n\u003Ctd>6.14 GiB\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q5_K_M\u003C\u002Ftd>\n\u003Ctd>Q5_K: 8 blocks of 32 weights, 6-bit scales and minimums\u003C\u002Ftd>\n\u003Ctd>5.5\u003C\u002Ftd>\n\u003Ctd>5.7036\u003C\u002Ftd>\n\u003Ctd>5.33 GiB\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q4_K_M\u003C\u002Ftd>\n\u003Ctd>Q4_K: same structure as Q5_K, 4-bit values\u003C\u002Ftd>\n\u003Ctd>4.5\u003C\u002Ftd>\n\u003Ctd>4.8944\u003C\u002Ftd>\n\u003Ctd>4.58 GiB\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>IQ4_XS\u003C\u002Ftd>\n\u003Ctd>Super-blocks of 256 weights, importance matrix\u003C\u002Ftd>\n\u003Ctd>4.25\u003C\u002Ftd>\n\u003Ctd>4.4597\u003C\u002Ftd>\n\u003Ctd>4.17 GiB\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>F16\u003C\u002Ftd>\n\u003Ctd>16-bit float\u003C\u002Ftd>\n\u003Ctd>16\u003C\u002Ftd>\n\u003Ctd>16.0005\u003C\u002Ftd>\n\u003Ctd>14.96 GiB\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003C\u002Fdiv>\n\u003Cp>The middle column comes from the \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Fhub\u002Fgguf\" rel=\"noopener\" class=\"ext\">Hugging Face GGUF docs\u003C\u002Fa>. The right-hand columns are measurements on Meta’s Llama 3.1 8B published in the \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fblob\u002Fmaster\u002Ftools\u002Fquantize\u002FREADME.md\" rel=\"noopener\" class=\"ext\">llama.cpp quantize README\u003C\u002Fa>. The half bit between “4” and Q4_K’s 4.5 pays for the scales and minimums. The measured Q4_K_M file adds another 0.39 bits, which suggests how much the higher-precision tensors in the mixture cost.\u003C\u002Fp>\n\u003Cp>You can switch the mixtures off. \u003Ccode>llama-quantize --pure\u003C\u002Fcode> “quantizes all tensors to the same type”.\u003C\u002Fp>\n\u003Ch2 id=\"the-size-arithmetic-worked-for-a-27b-model\">The size arithmetic, worked for a 27B model\u003C\u002Fh2>\n\u003Cp>The estimate is one line:\u003C\u002Fp>\n\u003Cdiv class=\"code-block\" data-lang=\"text\">\u003Cspan class=\"sr-only\">text code\u003C\u002Fspan>\u003Cpre tabindex=\"0\">\u003Ccode class=\"language-text\">file size in bytes ≈ parameters × bits per weight ÷ 8\n\u003C\u002Fcode>\u003C\u002Fpre>\u003C\u002Fdiv>\n\u003Cp>Take Qwen3.8-27B, the base of the \u003Ca href=\"\u002Fmodels\u002Fqwen3-8-27b-uncensored\u002F\">refusal-reduced Qwen3.8 27B variants\u003C\u002Fa> on this site. Hugging Face counts about 27.8 billion parameters in the original checkpoint, which accepts images and text. The text-model GGUF files in \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fhuihui-ai\u002FHuihui-Qwen3.8-27B-abliterated-GGUF\" rel=\"noopener\" class=\"ext\">huihui-ai’s GGUF repo\u003C\u002Fa> report 27,320,697,856 parameters, with the vision encoder in a separate 0.93 GB \u003Ccode>mmproj\u003C\u002Fcode> file. The difference is consistent with that file’s size at 16 bits per weight.\u003C\u002Fp>\n\u003Cdiv class=\"code-block\" data-lang=\"text\">\u003Cspan class=\"sr-only\">text code\u003C\u002Fspan>\u003Cpre tabindex=\"0\">\u003Ccode class=\"language-text\">At Q4_K's 4.5 bpw:          27,320,697,856 × 4.5    ÷ 8 ≈ 15.37 GB\nAt the measured 4.8944 bpw: 27,320,697,856 × 4.8944 ÷ 8 ≈ 16.71 GB\n\u003C\u002Fcode>\u003C\u002Fpre>\u003C\u002Fdiv>\n\u003Cp>The repo’s file labelled \u003Ccode>Q4_K\u003C\u002Fcode> (llama.cpp treats \u003Ccode>Q4_K\u003C\u002Fcode> as an alias for Q4_K_M) is 16,810,714,400 bytes, or 16.81 GB. Working backwards gives 4.92 bits per weight, close to the estimate.\u003C\u002Fp>\n\u003Cp>Real files rarely match the formula exactly, for three reasons:\u003C\u002Fp>\n\u003Cul>\n\u003Cli>Different tensors get different types, and the mixture depends on the architecture and on choices made by the person quantizing.\u003C\u002Fli>\n\u003Cli>Embedding and output tensors can be given their own types; \u003Ccode>llama-quantize\u003C\u002Fcode> has an option for each, and bartowski’s \u003Ccode>_L\u003C\u002Fcode> files use them.\u003C\u002Fli>\n\u003Cli>Metadata such as the tokenizer adds a small amount.\u003C\u002Fli>\n\u003C\u002Ful>\n\u003Cp>In the examples on this page, the estimate lands within a few percent of the real file. For the memory you need to run the file, which is more than the file size, use the \u003Ca href=\"\u002Ftools\u002Fvram-estimator\u002F\">VRAM estimator\u003C\u002Fa>.\u003C\u002Fp>\n\u003Ch2 id=\"worked-example-a-real-file-list\">Worked example: a real file list\u003C\u002Fh2>\n\u003Cp>The \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002Fbartowski\u002FLlama-3.3-70B-Instruct-abliterated-GGUF\" rel=\"noopener\" class=\"ext\">Llama 3.3 70B abliterated GGUF repo\u003C\u002Fa> quantizes huihui-ai’s abliterated Llama 3.3 70B. Byte counts below come from the Hugging Face API on 14 November 2026. Effective bits per weight is bytes × 8 ÷ 70,553,706,560, the parameter count Hugging Face reports for these files.\u003C\u002Fp>\n\u003Cdiv class=\"table-wrap\" role=\"region\" aria-label=\"Scrollable table\" tabindex=\"0\">\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Quant\u003C\u002Fth>\n\u003Cth>Bytes (shards added)\u003C\u002Fth>\n\u003Cth>GB\u003C\u002Fth>\n\u003Cth>GiB\u003C\u002Fth>\n\u003Cth>Effective bits per weight\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Q8_0 (2 shards)\u003C\u002Ftd>\n\u003Ctd>74,975,054,816\u003C\u002Ftd>\n\u003Ctd>74.98\u003C\u002Ftd>\n\u003Ctd>69.83\u003C\u002Ftd>\n\u003Ctd>8.50\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q6_K (2 shards)\u003C\u002Ftd>\n\u003Ctd>57,888,148,768\u003C\u002Ftd>\n\u003Ctd>57.89\u003C\u002Ftd>\n\u003Ctd>53.91\u003C\u002Ftd>\n\u003Ctd>6.56\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q5_K_M (2 shards)\u003C\u002Ftd>\n\u003Ctd>49,949,822,240\u003C\u002Ftd>\n\u003Ctd>49.95\u003C\u002Ftd>\n\u003Ctd>46.52\u003C\u002Ftd>\n\u003Ctd>5.66\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q4_K_M\u003C\u002Ftd>\n\u003Ctd>42,520,398,912\u003C\u002Ftd>\n\u003Ctd>42.52\u003C\u002Ftd>\n\u003Ctd>39.60\u003C\u002Ftd>\n\u003Ctd>4.82\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>IQ4_XS\u003C\u002Ftd>\n\u003Ctd>37,902,666,816\u003C\u002Ftd>\n\u003Ctd>37.90\u003C\u002Ftd>\n\u003Ctd>35.30\u003C\u002Ftd>\n\u003Ctd>4.30\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003C\u002Fdiv>\n\u003Cp>Three things stand out. The effective rates stay within 0.16 bits of llama.cpp’s 8B measurements. Several of the larger files are split into two shards, so add the parts before comparing sizes. And the card says every file was made with llama.cpp release b4381 using an imatrix built from a linked dataset; the 24.9 MB \u003Ccode>.imatrix\u003C\u002Fcode> file sits in the repo next to the quants.\u003C\u002Fp>\n\u003Cp>The model page for \u003Ca href=\"\u002Fmodels\u002Fllama-3-3-70b-abliterated\u002F\">Llama 3.3 70B abliterated\u003C\u002Fa> covers the variant itself.\u003C\u002Fp>\n\u003Ch2 id=\"gb-or-gib-check-the-bytes\">GB or GiB? Check the bytes\u003C\u002Fh2>\n\u003Cp>The \u003Ca href=\"https:\u002F\u002Fhuggingface.co\u002FHauhauCS\u002FGLM-4.7-Flash-Uncensored-HauhauCS-Balanced\" rel=\"noopener\" class=\"ext\">HauhauCS GLM-4.7-Flash card\u003C\u002Fa> lists FP16 at 56 GB, Q8_0 at 30 GB, Q6_K at 23 GB and Q4_K_M at 17 GB. The byte counts from the Hugging Face API tell a slightly different story:\u003C\u002Fp>\n\u003Cdiv class=\"table-wrap\" role=\"region\" aria-label=\"Scrollable table\" tabindex=\"0\">\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Quant\u003C\u002Fth>\n\u003Cth>Card says\u003C\u002Fth>\n\u003Cth>Bytes\u003C\u002Fth>\n\u003Cth>GB\u003C\u002Fth>\n\u003Cth>GiB\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>FP16\u003C\u002Ftd>\n\u003Ctd>56 GB\u003C\u002Ftd>\n\u003Ctd>59,908,836,928\u003C\u002Ftd>\n\u003Ctd>59.91\u003C\u002Ftd>\n\u003Ctd>55.79\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q8_0\u003C\u002Ftd>\n\u003Ctd>30 GB\u003C\u002Ftd>\n\u003Ctd>31,842,799,168\u003C\u002Ftd>\n\u003Ctd>31.84\u003C\u002Ftd>\n\u003Ctd>29.66\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q6_K\u003C\u002Ftd>\n\u003Ctd>23 GB\u003C\u002Ftd>\n\u003Ctd>24,614,785,600\u003C\u002Ftd>\n\u003Ctd>24.61\u003C\u002Ftd>\n\u003Ctd>22.92\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q4_K_M\u003C\u002Ftd>\n\u003Ctd>17 GB\u003C\u002Ftd>\n\u003Ctd>18,132,721,216\u003C\u002Ftd>\n\u003Ctd>18.13\u003C\u002Ftd>\n\u003Ctd>16.89\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003C\u002Fdiv>\n\u003Cp>All four card figures match the GiB values, rounded. One GiB is 1.074 GB, so the gap is about 7 percent, which matters when the fit is tight. The bartowski card above uses decimal GB. When the margin is small, read the byte count on the file page.\u003C\u002Fp>\n\u003Cp>Against the 29.94 billion parameters Hugging Face reports for these GGUF files, the effective rates are 16.01, 8.51, 6.58 and 4.85 bits per weight. Hugging Face counts 31.2 billion for the original GLM-4.7-Flash checkpoint, and the card does not say what accounts for the difference. Our \u003Ca href=\"\u002Fmodels\u002Fglm-4-7-flash-uncensored\u002F\">GLM-4.7-Flash variant page\u003C\u002Fa> covers the model itself.\u003C\u002Fp>\n\u003Ch2 id=\"quality-and-storage-tradeoffs\">Quality and storage tradeoffs\u003C\u002Fh2>\n\u003Cp>The quantize README is direct about the cost: quantization “may introduce some accuracy loss which is usually measured in Perplexity (ppl) and\u002For Kullback–Leibler Divergence (kld)”, and this “can be minimized by using a suitable imatrix file”.\u003C\u002Fp>\n\u003Cp>The only perplexity figures we cite are the help strings in llama.cpp’s \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fblob\u002Fmaster\u002Ftools\u002Fquantize\u002Fquantize.cpp\" rel=\"noopener\" class=\"ext\">quantize.cpp\u003C\u002Fa>, quoted exactly:\u003C\u002Fp>\n\u003Cdiv class=\"table-wrap\" role=\"region\" aria-label=\"Scrollable table\" tabindex=\"0\">\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Type\u003C\u002Fth>\n\u003Cth>Help string in quantize.cpp\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>Q8_0\u003C\u002Ftd>\n\u003Ctd>7.96G, +0.0026 ppl @ Llama-3-8B\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q6_K\u003C\u002Ftd>\n\u003Ctd>6.14G, +0.0217 ppl @ Llama-3-8B\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q5_K_M\u003C\u002Ftd>\n\u003Ctd>5.33G, +0.0569 ppl @ Llama-3-8B\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q4_K_M\u003C\u002Ftd>\n\u003Ctd>4.58G, +0.1754 ppl @ Llama-3-8B\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q4_K_S\u003C\u002Ftd>\n\u003Ctd>4.37G, +0.2689 ppl @ Llama-3-8B\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q3_K_M\u003C\u002Ftd>\n\u003Ctd>3.74G, +0.6569 ppl @ Llama-3-8B\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Q2_K\u003C\u002Ftd>\n\u003Ctd>2.96G, +3.5199 ppl @ Llama-3-8B\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>IQ4_XS\u003C\u002Ftd>\n\u003Ctd>4.25 bpw non-linear quantization (no perplexity given)\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003C\u002Fdiv>\n\u003Cp>Read them narrowly. They describe one 8B model. The strings do not name the evaluation text, the baseline perplexity or whether an imatrix was used. Perplexity measures next-token prediction on text, not accuracy on your task. What carries over is the shape: for this model, the penalty grows slowly from Q8_0 to Q5_K_M and much faster below 4 bits.\u003C\u002Fp>\n\u003Cp>We have not measured quality loss for any model on this site, and we have not tested whether quantization changes how often an \u003Ca href=\"\u002Flearn\u002Fwhat-is-abliteration\u002F\">abliterated model\u003C\u002Fa> refuses. Two practical rules do have a source. First, the quantize README warns that \u003Ccode>--allow-requantize\u003C\u002Fcode> “can severely reduce quality compared to quantizing from 16bit or 32bit”, so prefer files made from a 16-bit original. Second, the bartowski card suggests a file “1-2GB smaller than your GPU’s total VRAM” when you want everything on the GPU. That is the uploader’s rule of thumb, and long contexts need more headroom.\u003C\u002Fp>\n\u003Ch2 id=\"what-an-imatrix-does\">What an imatrix does\u003C\u002Fh2>\n\u003Cp>An importance matrix (imatrix) is computed by running the model over a file of calibration text. The \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fblob\u002Fmaster\u002Ftools\u002Fimatrix\u002FREADME.md\" rel=\"noopener\" class=\"ext\">imatrix README\u003C\u002Fa> describes the tool as computing “an importance matrix for a model and given text dataset”, to be used “during quantization to enhance the quality of the quantized models”. You pass the result to \u003Ccode>llama-quantize --imatrix\u003C\u002Fcode>.\u003C\u002Fp>\n\u003Cp>I-quants rely on it by design. K-quants and legacy quants can use it too. The calibration text is the quantizer’s choice, so a card that names its dataset, as bartowski’s does, tells you more than one that does not.\u003C\u002Fp>\n\u003Ch2 id=\"running-a-gguf-file\">Running a GGUF file\u003C\u002Fh2>\n\u003Cp>llama.cpp’s \u003Ccode>llama-server\u003C\u002Fcode> can fetch one quant directly from a Hugging Face repo with \u003Ccode>-hf\u003C\u002Fcode> (it defaults to Q4_K_M) or load a local file with \u003Ccode>-m\u003C\u002Fcode>. By default it listens on \u003Ccode>127.0.0.1:9931\u003C\u002Fcode>, per the \u003Ca href=\"https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fblob\u002Fmaster\u002Ftools\u002Fserver\u002FREADME.md\" rel=\"noopener\" class=\"ext\">server README\u003C\u002Fa>.\u003C\u002Fp>\n\u003Cp>To fetch one quant from Hugging Face and serve it:\u003C\u002Fp>\n\u003Cdiv class=\"code-block\" data-lang=\"bash\">\u003Cspan class=\"sr-only\">bash code\u003C\u002Fspan>\u003Cpre tabindex=\"0\">\u003Ccode class=\"language-bash\">llama-server -hf bartowski\u002FLlama-3.3-70B-Instruct-abliterated-GGUF:Q4_K_M -c 8192\n\u003C\u002Fcode>\u003C\u002Fpre>\u003C\u002Fdiv>\n\u003Cp>To load a file you already downloaded, with all layers on the GPU:\u003C\u002Fp>\n\u003Cdiv class=\"code-block\" data-lang=\"bash\">\u003Cspan class=\"sr-only\">bash code\u003C\u002Fspan>\u003Cpre tabindex=\"0\">\u003Ccode class=\"language-bash\">llama-server -m .\u002FGLM-4.7-Flash-Uncensored-HauhauCS-Balanced-Q4_K_M.gguf -c 16384 -ngl all\n\u003C\u002Fcode>\u003C\u002Fpre>\u003C\u002Fdiv>\n\u003Cp>Either way, the server answers on an OpenAI-compatible endpoint:\u003C\u002Fp>\n\u003Cdiv class=\"code-block\" data-lang=\"bash\">\u003Cspan class=\"sr-only\">bash code\u003C\u002Fspan>\u003Cpre tabindex=\"0\">\u003Ccode class=\"language-bash\">curl http:\u002F\u002F127.0.0.1:9931\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H "Content-Type: application\u002Fjson" \\\n  -d '{"messages":[{"role":"user","content":"Summarize this changelog in three bullet points: ..."}]}'\n\u003C\u002Fcode>\u003C\u002Fpre>\u003C\u002Fdiv>\n\u003Cp>\u003Ccode>-c\u003C\u002Fcode> sets the context size, and the KV cache for that context needs memory on top of the file. Check the card for runtime notes: the HauhauCS GLM card, for example, says the model is “Not recommended for Ollama due to chat template issues”. For choosing between running locally and calling an API, see \u003Ca href=\"\u002Flearn\u002Flocal-vs-cloud-llm\u002F\">local vs hosted LLMs\u003C\u002Fa>. \u003Ca href=\"\u002Fmodels\u002Flocal\u002F\">Models that run locally\u003C\u002Fa> lists variants with GGUF files.\u003C\u002Fp>\n\u003Ch2 id=\"glossary\">Glossary\u003C\u002Fh2>\n\u003Cdiv class=\"table-wrap\" role=\"region\" aria-label=\"Scrollable table\" tabindex=\"0\">\n\u003Ctable>\n\u003Cthead>\n\u003Ctr>\n\u003Cth>Term\u003C\u002Fth>\n\u003Cth>Meaning\u003C\u002Fth>\n\u003C\u002Ftr>\n\u003C\u002Fthead>\n\u003Ctbody>\n\u003Ctr>\n\u003Ctd>GGUF\u003C\u002Ftd>\n\u003Ctd>Single-file model format from the ggml project, holding tensors and key-value metadata.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>ggml\u003C\u002Ftd>\n\u003Ctd>The project whose repository hosts the GGUF specification; GGUF targets “GGML and executors based on GGML”.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Tensor\u003C\u002Ftd>\n\u003Ctd>One named array of weights, such as an attention projection in one layer.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Bits per weight (bpw)\u003C\u002Ftd>\n\u003Ctd>Storage per parameter, including scales. File bytes × 8 ÷ parameters.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Block, super-block\u003C\u002Ftd>\n\u003Ctd>A group of weights sharing a scale. K-quants and i-quants group blocks into super-blocks.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>K-quant\u003C\u002Ftd>\n\u003Ctd>Quant types such as Q4_K and Q6_K that quantize the block scales too.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>I-quant\u003C\u002Ftd>\n\u003Ctd>Quant types such as IQ4_XS, decoded using an importance matrix.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>\u003Ccode>_S\u003C\u002Fcode>, \u003Ccode>_M\u003C\u002Fcode>, \u003Ccode>_L\u003C\u002Fcode>\u003C\u002Ftd>\n\u003Ctd>Mixture recipes that store selected tensors at a higher-precision type.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>imatrix\u003C\u002Ftd>\n\u003Ctd>Importance matrix computed from calibration text and used during quantization.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Calibration text\u003C\u002Ftd>\n\u003Ctd>The text sample used to compute the imatrix.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Requantizing\u003C\u002Ftd>\n\u003Ctd>Quantizing an already-quantized file. The README warns it can severely reduce quality.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>mmproj\u003C\u002Ftd>\n\u003Ctd>Sidecar GGUF holding a vision or audio encoder for a multimodal model.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Shard\u003C\u002Ftd>\n\u003Ctd>One part of a model split across several files, named \u003Ccode>-0000N-of-0000M\u003C\u002Fcode>.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>Perplexity\u003C\u002Ftd>\n\u003Ctd>How well a model predicts a text; lower is better. A proxy, not task accuracy.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>KL divergence\u003C\u002Ftd>\n\u003Ctd>How far a quantized model’s token probabilities drift from the original’s.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>F16, BF16\u003C\u002Ftd>\n\u003Ctd>Unquantized 16-bit floating-point formats.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003Ctr>\n\u003Ctd>GB, GiB\u003C\u002Ftd>\n\u003Ctd>10⁹ bytes and 2³⁰ bytes. 1 GiB is about 1.074 GB.\u003C\u002Ftd>\n\u003C\u002Ftr>\n\u003C\u002Ftbody>\n\u003C\u002Ftable>\n\u003C\u002Fdiv>\n",[17,21,24,27,30,33,36,39,42,45],{"id":18,"text":19,"level":20},"what-a-gguf-file-is","What a GGUF file is",2,{"id":22,"text":23,"level":20},"how-to-read-a-quantization-name","How to read a quantization name",{"id":25,"text":26,"level":20},"bits-per-weight-the-type-vs-the-whole-file","Bits per weight: the type vs the whole file",{"id":28,"text":29,"level":20},"the-size-arithmetic-worked-for-a-27b-model","The size arithmetic, worked for a 27B model",{"id":31,"text":32,"level":20},"worked-example-a-real-file-list","Worked example: a real file list",{"id":34,"text":35,"level":20},"gb-or-gib-check-the-bytes","GB or GiB? Check the bytes",{"id":37,"text":38,"level":20},"quality-and-storage-tradeoffs","Quality and storage tradeoffs",{"id":40,"text":41,"level":20},"what-an-imatrix-does","What an imatrix does",{"id":43,"text":44,"level":20},"running-a-gguf-file","Running a GGUF file",{"id":46,"text":47,"level":20},"glossary","Glossary",1984,[50,54,58,62,65,68,71,74,78,82],{"title":51,"publisher":52,"url":53},"GGUF specification","GitHub (ggml-org\u002Fggml)","https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fggml\u002Fblob\u002Fmaster\u002Fdocs\u002Fgguf.md",{"title":55,"publisher":56,"url":57},"GGUF","Hugging Face Hub documentation","https:\u002F\u002Fhuggingface.co\u002Fdocs\u002Fhub\u002Fgguf",{"title":59,"publisher":60,"url":61},"llama.cpp quantize README","GitHub (ggml-org\u002Fllama.cpp)","https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fblob\u002Fmaster\u002Ftools\u002Fquantize\u002FREADME.md",{"title":63,"publisher":60,"url":64},"quantize.cpp (quantization type list)","https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fblob\u002Fmaster\u002Ftools\u002Fquantize\u002Fquantize.cpp",{"title":66,"publisher":60,"url":67},"k-quants (pull request #1684)","https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fpull\u002F1684",{"title":69,"publisher":60,"url":70},"llama.cpp imatrix README","https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fblob\u002Fmaster\u002Ftools\u002Fimatrix\u002FREADME.md",{"title":72,"publisher":60,"url":73},"LLaMA.cpp HTTP Server README","https:\u002F\u002Fgithub.com\u002Fggml-org\u002Fllama.cpp\u002Fblob\u002Fmaster\u002Ftools\u002Fserver\u002FREADME.md",{"title":75,"publisher":76,"url":77},"Llama-3.3-70B-Instruct-abliterated-GGUF","Hugging Face (bartowski)","https:\u002F\u002Fhuggingface.co\u002Fbartowski\u002FLlama-3.3-70B-Instruct-abliterated-GGUF",{"title":79,"publisher":80,"url":81},"GLM-4.7-Flash-Uncensored-HauhauCS-Balanced","Hugging Face (HauhauCS)","https:\u002F\u002Fhuggingface.co\u002FHauhauCS\u002FGLM-4.7-Flash-Uncensored-HauhauCS-Balanced",{"title":83,"publisher":84,"url":85},"Huihui-Qwen3.8-27B-abliterated-GGUF","Hugging Face (huihui-ai)","https:\u002F\u002Fhuggingface.co\u002Fhuihui-ai\u002FHuihui-Qwen3.8-27B-abliterated-GGUF",[87,90,93,96],{"q":88,"a":89},"Which GGUF quant should I download?","Start from your memory budget, not the name. Pick the largest file that leaves room for the KV cache and compute buffers, then test it on your own prompts against a larger quant if you can.",{"q":91,"a":92},"Is Q4_K_M a 4-bit file?","Not exactly. The Q4_K type stores 4.5 bits per weight, and Q4_K_M mixes in higher-precision types for some tensors. llama.cpp measured 4.89 bits per weight for a whole Llama 3.1 8B file.",{"q":94,"a":95},"Why does a model card list a different size from what I downloaded?","Often units. Some cards round GiB and call it GB. A file of 18,132,721,216 bytes is 18.13 GB but 16.89 GiB.",{"q":97,"a":98},"Does quantization change how often an abliterated model refuses?","We have not tested this and found no source that measures it. Compare the quant you plan to use against a higher-precision file on your own prompts.",[100,104,108,112],{"path":101,"title":102,"description":103},"\u002Flearn\u002Flocal-vs-cloud-llm\u002F","Local vs hosted uncensored LLMs","Run a refusal-reduced model yourself or call a hosted API? Privacy, latency, upkeep and a worked cost comparison in which every input is a stated assumption.",{"path":105,"title":106,"description":107},"\u002Ftools\u002Fvram-estimator\u002F","VRAM estimator","Weights, KV cache and overhead for local inference.",{"path":109,"title":110,"description":111},"\u002Fmodels\u002Flocal\u002F","Models for local use","GGUF and other local formats with memory estimates.",{"path":113,"title":114,"description":115},"\u002Flearn\u002Fwhat-is-abliteration\u002F","What is abliteration?","Abliteration edits an LLM's weights to suppress the activation direction tied to refusals. How it works conceptually, how it is measured and its limits.",null,[118,105,119,120,113,101,109],"\u002Fmodels\u002Fqwen3-8-27b-uncensored\u002F","\u002Fmodels\u002Fllama-3-3-70b-abliterated\u002F","\u002Fmodels\u002Fglm-4-7-flash-uncensored\u002F",1794622336927]