--- license: apache-2.0 base_model: - Qwen/Qwen3.8-27B tags: - qwen3.8 - thinking - reasoning - gguf - quantized - imatrix - mtp - llama.cpp - agentic - coding - conversational pipeline_tag: image-text-to-text --- # Quality-first GGUF quants of Qwen3.8-27B **Consider this fine-tune instead, it is even better:** [Kiwen1.2-27B](https://huggingface.co/NikiKrutan/Kiwen1.2-27B-MTP-GGUF) **In the process of creating these quants:** - **95 hours** of 2xGPU near 100% load were spent! - **27 TB** of not so good ggufs were materialized, measured and put to trash *(all were usable)*! - **72 man-hours** were spent! These quants were produced by my quality-first experimental tool, **niki-allocator**. The main difference of my method is per-tensor type allocation (not per tensor groups, layers, community lore etc.) Each tensor gets what is *estimated* to be *best* for it. After that best allocations are tested with real KLD runs to choose the winner. [Source model](https://huggingface.co/Qwen/Qwen3.8-27B) ## Quantization quality comparison To prove my method I've performed comparison with quants from other providers: [unsloth](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF), [AtomicChat](https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF), [bartowski](https://huggingface.co/bartowski/Qwen3.8-27B-GGUF), [lmstudio-community](https://huggingface.co/lmstudio-community/Qwen3.8-27B-GGUF), [mudler](https://huggingface.co/mudler/Qwen3.8-27B-APEX-GGUF), [byteshape](https://huggingface.co/byteshape/Qwen3.8-27B-GGUF) ***Lower is better*** ### Main metrics ![Mean KLD, niki-small-plus-2 corpus](plots/niki_mean_kld.png) ![Mean KLD, independent-v2 corpus](plots/indep_mean_kld.png) ![RMS Δp, niki-small-plus-2 corpus](plots/niki_rms_dp.png) ![RMS Δp, independent-v2 test corpus](plots/indep_rms_dp.png) ### Tail metrics
*p99.9 are extremely corpus-driven with usual test corpus sizes of ~300-400K tokens* ![p95 KLD, niki-small-plus-2 corpus](plots/niki_p95.png) ![p95 KLD, independent-v2 corpus](plots/indep_p95.png) ![p99 KLD, niki-small-plus-2 corpus](plots/niki_p99.png) ![p99 KLD, independent-v2 corpus](plots/indep_p99.png) ![p99.9 KLD, niki-small-plus-2 corpus](plots/niki_p99_9.png) ![p99.9 KLD, independent-v2 corpus](plots/indep_p99_9.png)
### Same top probability *Useful for temperature 0, otherwise RMS Δp should be considered.*
Note that graphs show **1 - same top p**, so lower is better. ![Same top probability, niki-small-plus-2 corpus](plots/niki_same_top_p.png) ![Same top probability, independent-v2 corpus](plots/indep_same_top_p.png)
### Perplexity (PPL) *Very crude noisy metric, cheaper to compute though, that's why so popular*
![PPL, niki-small-plus-2 corpus](plots/niki_ppl.png) ![PPL, independent-v2 corpus](plots/indep_ppl.png)
### Notes *TL;DR **difference of corpora** for end user*: `niki-small-plus-2` - coding, agentic, reasoning, hard logic prompts, English, Russian; `independent-v2` - multilingual, English, web texts, science papers, math, legal, no reasoning or agentic traces, no code. `Unsloth 3.0 preview`, `AtomicChat`, `bartowski`, `lmstudio-community` quants for comparison were downloaded on **August 16, 2026.** `Unsloth 3.0`, `mudler`, `byteshape` quants for comparison were downloaded on **August 21, 2026.** I've deliberately dropped some known bad quants (legacy quant types and such) to save my time on downloading and measuring. Unsloth v3.0 was introduced *after* I finished my work. I added them for clean honest comparison. I don't like their marketing attitude and evidence presentation *(just drop in their best graph without details on how exactly it was measured and if it is meaningful metric for end user)*. But after all they provided really good quants (mostly). Unsloth v3.0 dropped MTP for lower quants reasoning that they are not needed. Well, I can't agree. From my findings MTP works well on lower quants. I added these no-MTP quants to comparison, but added fixed-size correction of their MTP v3.0 fixed-choice for all their non-MTP quants. Yeah, I know.. So for *"honest"* *scientific* comparison of only target models without strange choices of MTP quantization, here is comparison for all quants where only size of target model is accounted. This is not that *honest* for the end user though. End user downloads quant and either fits it in his VRAM/RAM or not, poor choice of oversized MTP can't be excuse here. Quant provider should provide MTP quantization based on their research, not just drop in something, or just delete it. It is part of a quant. But here you go: ### All graphs with MTP size subtracted
### Main metrics ![Mean KLD, niki-small-plus-2 corpus](plots-nomtp/niki_mean_kld.png) ![Mean KLD, independent-v2 corpus](plots-nomtp/indep_mean_kld.png) ![RMS Δp, niki-small-plus-2 corpus](plots-nomtp/niki_rms_dp.png) ![RMS Δp, independent-v2 test corpus](plots-nomtp/indep_rms_dp.png) ### Tail metrics *p99.9 are extremely corpus-driven with usual test corpus sizes of ~300-400K tokens* ![p95 KLD, niki-small-plus-2 corpus](plots-nomtp/niki_p95.png) ![p95 KLD, independent-v2 corpus](plots-nomtp/indep_p95.png) ![p99 KLD, niki-small-plus-2 corpus](plots-nomtp/niki_p99.png) ![p99 KLD, independent-v2 corpus](plots-nomtp/indep_p99.png) ![p99.9 KLD, niki-small-plus-2 corpus](plots-nomtp/niki_p99_9.png) ![p99.9 KLD, independent-v2 corpus](plots-nomtp/indep_p99_9.png) ### Same top probability *Useful for temperature 0, otherwise RMS Δp should be considered.* Note that graphs show **1 - same top p**, so lower is better. ![Same top probability, niki-small-plus-2 corpus](plots-nomtp/niki_same_top_p.png) ![Same top probability, independent-v2 corpus](plots-nomtp/indep_same_top_p.png) ### Perplexity (PPL) *Very crude noisy metric, cheaper to compute though, that's why so popular* ![PPL, niki-small-plus-2 corpus](plots-nomtp/niki_ppl.png) ![PPL, independent-v2 corpus](plots-nomtp/indep_ppl.png)
***See the tables below for exact numbers for all quants and metrics*** ## Sizes I've chosen 15 sizes to fill 8500-23000 MiB with geometric progression and rounded to 100 MiB *niki-allocator is able to produce allocation with ~1 MiB precision.* ### Which one to choose? If you are after *quality*: ***the bigger - the better***. So it is always a compromise of what you can afford considering context size and other constraints. Choose the biggest you can fit.
About speed I haven't measured *speed* since it is highly hardware and configuration dependable. What is faster for me may be slower for you. If the speed is the only that matters and you have Blackwell - get some `NVFP4` quant (but quality is considerably worse for the same quant size). Otherwise you should measure speed yourself on your hardware and your configuration to make real decision. By the way, on my setup `ubatch` is the parameter that has huge impact on speed. On 27B models 384 was found to be optimal in my case (2x5060Ti), 512 or 256 are much slower. So probably you should test this first.
### Quant names Hugging Face requires quant type in the filename to properly show it. But all my quants are mixed by design. I decided to include dominating quant type (by size, not by tensor count) to the name therefore. It may easily be less than 50% of the model though. So **choose by size, ignore listed quant type**. For my quants: bigger --> better quality. Always. ## Methodology ### niki-allocator
What is niki-allocator It is quality-first per-tensor quantization allocator for `llama.cpp` llama-quantize. Allocator decides, per tensor, which quant type to use so that the resulting GGUF hits an exact target file size with the best *estimated* quality. It is not direct measurement of KLD-metrics, it is only estimation. But as you can see from real KLD evaluations estimation is good enough. Raw results are already good. But I added some calibration levers on top to make it better (for [Qwen3.5-0.8B](https://huggingface.co/NikiKrutan/Qwen3.5-0.8B-MTP-GGUF) I used uncalibrated estimation). These levers need series of KLD evaluations to move estimation closer to reality, so it takes hours-days of GPU time on my consumer setup. On top of this I added bayesian search seeded with best results from levers to move calibration even further. This is very GPU-intensive, literally took days (nights actually) of GPU time on my setup.
### Corpora
Corpora choice, differences, details - `imatrix`: `niki-small-2` corpus - ~330K tokens from my own agentic sessions (English, Russian, design docs with hard logic, code, reasoning). Deduped, cleaned, filtered by simhash similarity to include only really different turns. Turn-aligned by tokenizer so that each chunk contains exactly 1 turn. Short turns dropped. Used for `imatrix` only. - Calibration of `niki-allocator`: hardest 16 chunks of `niki-small-plus-2` for faster crude KLD calibration (full corpus is too slow on my hardware). - Evaluation 1: `niki-small-plus-2` corpus - ~400K tokens from my own agentic sessions (English, Russian, design docs with hard logic, code, reasoning). Deduped, cleaned, filtered by simhash similarity to include only really different turns. Turn-aligned by tokenizer so that each chunk contains exactly 1 turn. Short turns dropped. Does not include any exact or similar (by simhash) agentic turns from `niki-small`. So while it is from the same domain as imatrix-corpus, it is different. - Evaluation 2 (truly *independent* this time): `independent-v2` corpus - ~390 K tokens. This corpus is random compilation of top-13 languages prose (`HuggingFaceFW/fineweb-2`) excluding English (it is everywhere in other domains already), legal domain (`common-pile/caselaw_access_project`), math domain (`open-web-math/open-web-math`), science domain (`recursal/arXiv-CC0-v0.5`). *Code and agentic data wasn't included since my quants are highly tuned for these already (it wouldn't be honest comparison).* **This dataset wasn't seen during calibration or imatrix at all.** Used only for final evaluation before publishing. So my quants are not anyhow tuned for this dataset. Foreign quants haven't seen this exact dataset as well (random pick). Notes: - `independent-v2` corpus will be published [here](https://huggingface.co/datasets/NikiKrutan/independent). *Note that you can't redistribute it without proper attributions due to license of some included papers.* - `niki-*` corpora are not published. There is nothing really special actually. They just contain my private information. Sorry for inconvenience. - Why not `wikitext-2 test`? While wiki may be considered as independent evaluation corpus for my quants. It easily may not be true for others quants (wikitext-2 is the most popular corpus somehow for such jobs). They may have seen it (or part of it) during imatrix creation or their methodology calibration. So for really ***independent*** evaluation there should be corpus that wasn't seen anyhow by any quants and even better if it wasn't seen during base model creation. Also `wiki` is mostly homogenous (not very easy, but there are very few hard tokens and chunks). My corpus is everything from "walk in the park" to "high mountain climbing". Its tail metrics are much higher. It is actually more representative for the real model work (if crawling wiki is not your main use case of course). So I use my own corpus because I need these quants to be good in my tasks. And `independent-v2` as truly independent evaluation.
### Chunk sizes
- `imatrix` (`niki-small-2`): 8064 tokens (384 x 21). 41 chunks total. To catch reasoning and long turn traces, not just basic chat. - `niki-small-plus-2` evaluation: 6144 tokens (384 x 16). 65 chunks total. To catch reasoning and long turns, but to keep chunks amount somewhat reasonable (that's why lower than for imatrix). - `independent-v2` evaluation: 2304 tokens (384 x 6). 171 chunks total. Long enough for chat and basic tasks. *Why 384 magic number?* *My hardware is fastest with `ubatch` 384, so I use it.*
### Metrics All metrics were evaluated with `llama-perplexity` against **Q8_0** standard quant (due to hardware constraints). Quantization was performed with patched `llama-quantize` from **BF_16** source. Fully compatible with original `llama.cpp`. All my and foreign quants were evaluated against the same base. `llama.cpp` commit `876a4321163249c43ca4e986818fab5ab081f282`. The main metrics during calibration of `niki-allocator` were `mean KLD` and `RMS Δp`. Better `PPL` doesn't always mean better other more important metrics. So it can't be used as a main metric *(some people still use)*. It is just fast indicator (no need for logits base) to check for big problems. It can't be main quality metric. Don't make or accept comparisons done only by PPL. It is noise. Notes on `p99 KLD`, `p99.9 KLD`, `max KLD`. With such a small corpora these metrics are not very robust. While `p99 KLD` can give some plausible comparison for the allocation method as a whole, `p99.9` is extremely corpus-driven (too little data), `max KLD` is just random at any corpus size (one hardest token of the whole dataset, saying nothing about other hard tokens). So I don't tune to these. And anybody who does with less than several M tokens corpus is just over-fitting to their exact corpus. *Please correct me if I am wrong.* ## Evaluation results ### niki-small-plus-2 corpus
Warning! Huge table inside! *Lower is better **except same top p***

Qwen3.8-27B — niki-small-plus-2 corpus

Source Label Size, MiB* Size w/o MTP, MiB Mean KLD RMS Δp KLD p95 KLD p99 KLD p99.9 Same top p, % PPL
unsloth 3.0 UD-IQ1_S 6240 * 5905 0.59681 ± 0.00410 21.210 ± 0.063 1.6709 10.2128 22.0603 75.44 ± 0.10 4.610 ± 0.026
unsloth 3.0 UD-IQ1_M 6752 * 6417 0.53205 ± 0.00457 18.591 ± 0.066 1.2365 12.3683 23.5830 78.62 ± 0.09 4.263 ± 0.025
unsloth 3.0 UD-IQ2_XXS 7264 * 6929 0.42809 ± 0.00430 15.342 ± 0.064 0.8708 11.1274 23.3418 81.79 ± 0.09 4.350 ± 0.028
AtomicChat AD-IQ1_M 8104 7826 0.40475 ± 0.00409 15.342 ± 0.064 0.8258 10.2505 23.0421 81.78 ± 0.09 3.905 ± 0.022
unsloth 3.0 UD-IQ2_S 8319 * 7984 0.35110 ± 0.00429 12.211 ± 0.069 0.5158 11.1857 23.2209 85.77 ± 0.08 4.331 ± 0.029
NikiKrutan NIKI-IQ2_XXS 8500 8316 0.29969 ± 0.00364 11.724 ± 0.062 0.4843 8.4590 21.2619 85.49 ± 0.08 4.065 ± 0.025
AtomicChat AD-IQ2_XXS 8561 8283 0.40775 ± 0.00449 13.996 ± 0.067 0.6582 12.1255 23.5798 83.37 ± 0.08 4.615 ± 0.032
unsloth 3.0 preview UD-IQ2_XXS 8593 8382 0.29882 ± 0.00369 11.384 ± 0.059 0.4793 8.3221 21.7652 85.56 ± 0.08 4.171 ± 0.027
bartowski IQ2_XXS 8958 8730 0.40331 ± 0.00404 14.430 ± 0.059 0.8150 10.3693 22.2155 81.90 ± 0.09 4.419 ± 0.029
NikiKrutan NIKI-IQ2_S 9100 8925 0.29060 ± 0.00385 10.310 ± 0.063 0.3783 9.0926 22.4354 87.09 ± 0.07 4.178 ± 0.028
unsloth 3.0 UD-Q2_K_XL 9374 9039 0.23685 ± 0.00343 9.208 ± 0.063 0.2930 6.8906 20.9554 88.65 ± 0.07 3.807 ± 0.023
AtomicChat AD-IQ2_XS 9432 9153 0.33312 ± 0.00420 11.466 ± 0.066 0.4421 10.7630 22.8707 86.09 ± 0.08 4.395 ± 0.030
bartowski IQ2_XS 9524 9296 0.38705 ± 0.00434 12.842 ± 0.062 0.6032 11.7030 22.9351 83.97 ± 0.08 4.470 ± 0.031
AtomicChat AD-IQ2_S-IQ2_XS 9748 9469 0.29551 ± 0.00394 10.421 ± 0.067 0.3584 9.7478 22.0738 87.27 ± 0.07 4.248 ± 0.029
NikiKrutan NIKI-IQ3_XXS 9800 9622 0.24855 ± 0.00365 8.648 ± 0.059 0.2652 8.1560 22.2199 88.86 ± 0.07 3.947 ± 0.025
bartowski IQ2_S 9818 9590 0.37596 ± 0.00436 12.518 ± 0.066 0.5408 11.8180 22.9918 84.70 ± 0.08 4.483 ± 0.031
unsloth 3.0 preview UD-IQ2_M 9842 9636 0.29107 ± 0.00401 9.528 ± 0.066 0.2816 10.4280 21.9791 87.91 ± 0.07 4.028 ± 0.026
unsloth 3.0 preview UD-Q2_K_XL 10182 9976 0.23467 ± 0.00349 9.201 ± 0.067 0.2472 7.3177 21.1131 88.66 ± 0.07 4.010 ± 0.025
bartowski IQ2_M 10370 10142 0.32379 ± 0.00419 10.543 ± 0.064 0.3837 10.7744 22.9631 86.66 ± 0.08 4.328 ± 0.030
unsloth 3.0 UD-IQ3_XXS 10428 10094 0.21417 ± 0.00346 7.863 ± 0.069 0.1830 7.0054 21.3873 90.48 ± 0.07 3.886 ± 0.024
NikiKrutan NIKI-IQ3_XXS 10500 10322 0.19533 ± 0.00309 8.268 ± 0.062 0.2035 5.4233 19.7968 89.92 ± 0.07 3.712 ± 0.021
AtomicChat AD-IQ2_S 10626 10347 0.24532 ± 0.00358 9.189 ± 0.066 0.2605 7.7219 21.6210 88.96 ± 0.07 4.076 ± 0.026
mudler APEX-I-Nano 10720 10289 0.28538 ± 0.00385 10.279 ± 0.070 0.3606 9.3696 21.9993 87.91 ± 0.07 4.286 ± 0.029
byteshape IQ3_S-3.44bpw 11226 11058 0.20084 ± 0.00331 7.754 ± 0.072 0.1755 6.1658 20.8096 91.00 ± 0.06 3.955 ± 0.025
NikiKrutan NIKI-IQ3_S 11300 11116 0.18301 ± 0.00312 6.935 ± 0.065 0.1429 5.7343 20.1450 91.42 ± 0.06 3.845 ± 0.024
unsloth 3.0 preview UD-IQ3_XXS 11362 11167 0.20210 ± 0.00335 7.402 ± 0.071 0.1454 6.8158 20.7976 91.19 ± 0.06 3.714 ± 0.022
unsloth 3.0 UD-IQ3_S 11483 11148 0.18327 ± 0.00331 6.523 ± 0.069 0.1247 5.5946 21.1550 92.14 ± 0.06 3.837 ± 0.024
AtomicChat AD-IQ3_XXS 11516 11238 0.19840 ± 0.00325 7.755 ± 0.068 0.1677 6.1414 20.4022 90.75 ± 0.07 3.934 ± 0.025
byteshape IQ4_XS-3.67bpw 11975 11806 0.17338 ± 0.00312 6.696 ± 0.072 0.1239 5.1036 20.5042 92.21 ± 0.06 3.728 ± 0.022
bartowski IQ3_XXS 12042 11814 0.19534 ± 0.00327 8.055 ± 0.070 0.1887 5.6009 20.9745 90.41 ± 0.07 3.779 ± 0.023
NikiKrutan NIKI-IQ3_S 12100 11890 0.17271 ± 0.00314 6.316 ± 0.069 0.1059 5.3612 20.4380 92.16 ± 0.06 3.740 ± 0.023
AtomicChat AD-IQ3_S-IQ3_XXS 12383 12104 0.20045 ± 0.00341 6.686 ± 0.072 0.1206 7.2896 20.8107 91.87 ± 0.06 3.799 ± 0.023
unsloth 3.0 UD-Q3_K_XL 12537 12203 0.17021 ± 0.00325 6.254 ± 0.079 0.0846 5.2435 20.9727 93.10 ± 0.06 3.712 ± 0.023
bartowski IQ3_XS 12713 12485 0.19216 ± 0.00319 7.144 ± 0.067 0.1438 6.3475 19.8683 91.29 ± 0.06 3.865 ± 0.024
unsloth 3.0 preview UD-Q3_K_XL 12818 12620 0.14791 ± 0.00288 6.059 ± 0.073 0.0843 4.0237 19.7203 93.02 ± 0.06 3.698 ± 0.022
NikiKrutan NIKI-IQ4_XS 13000 12790 0.14490 ± 0.00287 5.600 ± 0.071 0.0753 3.8287 19.8499 93.17 ± 0.06 3.757 ± 0.023
byteshape IQ4_XS-4.00bpw 13030 12835 0.15545 ± 0.00301 5.622 ± 0.071 0.0842 4.3696 20.4840 93.33 ± 0.06 3.813 ± 0.024
AtomicChat AD-IQ3_S 13197 12971 0.15641 ± 0.00298 5.882 ± 0.070 0.0908 4.4304 20.3319 92.90 ± 0.06 3.820 ± 0.024
bartowski IQ3_M 13259 13032 0.17815 ± 0.00308 7.029 ± 0.069 0.1270 5.3719 19.9435 91.88 ± 0.06 3.880 ± 0.024
mudler APEX-I-Mini 13297 12867 0.20655 ± 0.00349 7.053 ± 0.074 0.1328 7.2564 21.9309 91.81 ± 0.06 3.904 ± 0.025
unsloth 3.0 UD-IQ4_XS 13593 13258 0.14809 ± 0.00301 5.766 ± 0.085 0.0524 4.2925 20.2216 94.27 ± 0.05 3.622 ± 0.021
AtomicChat AD-IQ4_XS-IQ3_S 13769 13534 0.14858 ± 0.00294 5.325 ± 0.071 0.0713 4.2618 19.8525 93.39 ± 0.06 3.834 ± 0.024
NikiKrutan NIKI-IQ4_XS 14000 13790 0.12202 ± 0.00264 4.917 ± 0.075 0.0480 2.8676 18.7476 94.43 ± 0.05 3.696 ± 0.022
byteshape IQ4_XS-4.40bpw 14344 14115 0.13487 ± 0.00284 4.788 ± 0.077 0.0465 3.6616 19.6340 94.69 ± 0.05 3.777 ± 0.023
unsloth 3.0 UD-Q4_K_S 14647 14312 0.12428 ± 0.00274 4.637 ± 0.079 0.0370 2.8792 19.6644 95.15 ± 0.05 3.688 ± 0.022
bartowski IQ4_XS 14847 14619 0.12968 ± 0.00271 5.155 ± 0.081 0.0467 3.3971 18.9806 94.47 ± 0.05 3.905 ± 0.025
unsloth 3.0 preview IQ4_XS 14978 14725 0.11673 ± 0.00258 4.766 ± 0.078 0.0423 2.7687 18.5722 94.75 ± 0.05 3.863 ± 0.024
NikiKrutan NIKI-IQ4_XS 15000 14785 0.11079 ± 0.00253 4.354 ± 0.076 0.0362 2.3719 18.7210 95.06 ± 0.05 3.741 ± 0.023
byteshape Q5_K_S-4.72bpw 15394 15144 0.13078 ± 0.00281 4.478 ± 0.080 0.0356 3.5353 19.4105 95.07 ± 0.05 3.763 ± 0.023
unsloth 3.0 UD-Q4_K_M 15702 15367 0.11679 ± 0.00264 4.138 ± 0.078 0.0300 2.6716 19.2746 95.47 ± 0.05 3.809 ± 0.024
AtomicChat AD-IQ4_XS 15748 15490 0.11266 ± 0.00256 4.346 ± 0.078 0.0329 2.4330 18.9169 95.22 ± 0.05 3.769 ± 0.023
bartowski Q4_K_S 15939 15711 0.12847 ± 0.00276 4.698 ± 0.079 0.0399 3.3534 19.5019 94.76 ± 0.05 3.763 ± 0.023
lmstudio-community Q4_K_M 16032 15781 0.16945 ± 0.00319 6.184 ± 0.077 0.0850 5.4682 20.9170 93.02 ± 0.06 3.639 ± 0.021
NikiKrutan NIKI-IQ4_XS 16100 15885 0.10334 ± 0.00243 4.209 ± 0.076 0.0313 2.0672 18.0062 95.27 ± 0.05 3.758 ± 0.023
unsloth 3.0 preview Q4_K_M 16314 16038 0.12098 ± 0.00269 4.572 ± 0.079 0.0378 2.8831 19.4748 94.97 ± 0.05 3.769 ± 0.023
AtomicChat AD-Q4_K 16328 16062 0.11616 ± 0.00265 4.432 ± 0.082 0.0300 2.6443 19.0882 95.42 ± 0.05 3.703 ± 0.023
unsloth 3.0 UD-Q4_K_XL 16746 16411 0.11012 ± 0.00258 3.950 ± 0.082 0.0225 2.3869 18.8515 95.89 ± 0.04 3.770 ± 0.023
bartowski Q4_K_M 16949 16721 0.11468 ± 0.00262 4.188 ± 0.078 0.0320 2.5264 18.8423 95.23 ± 0.05 3.822 ± 0.024
unsloth 3.0 preview UD-Q4_K_XL 17093 16821 0.10611 ± 0.00252 3.925 ± 0.080 0.0230 2.2947 18.6469 95.88 ± 0.04 3.772 ± 0.023
NikiKrutan NIKI-Q5_K_S 17300 17085 0.09718 ± 0.00239 3.641 ± 0.079 0.0191 1.8434 18.0073 96.10 ± 0.04 3.780 ± 0.023
AtomicChat AD-Q5_K-Q4_K 17688 17411 0.09972 ± 0.00240 3.907 ± 0.081 0.0216 1.9442 18.0818 96.01 ± 0.04 3.730 ± 0.023
unsloth 3.0 UD-Q5_K_S 17801 17466 0.10612 ± 0.00251 3.671 ± 0.082 0.0170 2.3331 18.3369 96.18 ± 0.04 3.796 ± 0.024
bartowski Q4_K_L 17849 17621 0.11493 ± 0.00260 4.304 ± 0.079 0.0311 2.7333 18.9470 95.34 ± 0.05 3.822 ± 0.024
byteshape Q5_K_M-5.60bpw 18249 17965 0.09784 ± 0.00241 3.863 ± 0.085 0.0164 1.9265 17.9692 96.35 ± 0.04 3.715 ± 0.023
unsloth 3.0 preview Q5_K_S 18377 18080 0.10613 ± 0.00253 3.784 ± 0.082 0.0192 2.2872 18.5994 96.03 ± 0.04 3.739 ± 0.023
NikiKrutan NIKI-Q5_K_M 18600 18385 0.08612 ± 0.00224 3.372 ± 0.082 0.0135 1.4814 17.2961 96.57 ± 0.04 3.792 ± 0.024
bartowski Q5_K_S 18769 18541 0.09956 ± 0.00244 3.644 ± 0.081 0.0177 1.9769 18.5421 96.09 ± 0.04 3.798 ± 0.024
unsloth 3.0 UD-Q5_K_M 18856 18521 0.09175 ± 0.00233 3.449 ± 0.082 0.0132 1.6733 17.4314 96.60 ± 0.04 3.789 ± 0.024
unsloth 3.0 preview Q5_K_M 18915 18606 0.10288 ± 0.00251 3.644 ± 0.082 0.0172 2.0636 18.6094 96.23 ± 0.04 3.783 ± 0.024
unsloth 3.0 preview UD-Q5_K_XL 19282 18958 0.08932 ± 0.00230 3.423 ± 0.082 0.0135 1.5523 17.6310 96.52 ± 0.04 3.806 ± 0.024
AtomicChat AD-Q5_K 19295 18990 0.08701 ± 0.00225 3.311 ± 0.081 0.0126 1.4779 17.3852 96.63 ± 0.04 3.760 ± 0.023
bartowski Q5_K_M 19791 19563 0.09182 ± 0.00233 3.745 ± 0.086 0.0142 1.6026 17.6914 96.47 ± 0.04 3.757 ± 0.023
unsloth 3.0 UD-Q5_K_XL 19910 19575 0.09123 ± 0.00232 3.344 ± 0.084 0.0112 1.6803 17.6995 96.88 ± 0.04 3.790 ± 0.024
NikiKrutan NIKI-Q6_K 20000 19785 0.07126 ± 0.00200 3.306 ± 0.087 0.0085 1.0398 15.9125 97.17 ± 0.04 3.772 ± 0.024
bartowski Q5_K_L 20540 20312 0.08929 ± 0.00231 3.458 ± 0.084 0.0137 1.4471 17.7038 96.67 ± 0.04 3.779 ± 0.024
unsloth 3.0 UD-Q6_K 20965 20631 0.08346 ± 0.00219 3.176 ± 0.087 0.0086 1.3687 17.0122 97.13 ± 0.04 3.754 ± 0.023
lmstudio-community Q6_K 21392 21060 0.07748 ± 0.00211 3.277 ± 0.083 0.0105 1.1575 16.4780 96.89 ± 0.04 3.734 ± 0.023
NikiKrutan NIKI-Q6_K 21400 21185 0.05081 ± 0.00163 2.581 ± 0.083 0.0054 0.4680 13.4905 97.68 ± 0.03 3.770 ± 0.023
unsloth 3.0 preview Q6_K 21824 21480 0.05497 ± 0.00169 2.794 ± 0.082 0.0069 0.5595 13.8988 97.36 ± 0.04 3.790 ± 0.024
AtomicChat AD-Q6_K-Q5_K 22012 21689 0.08001 ± 0.00217 3.097 ± 0.083 0.0093 1.2233 16.9064 97.14 ± 0.04 3.772 ± 0.023
unsloth 3.0 UD-Q6_K_M 22019 21684 0.07816 ± 0.00211 3.066 ± 0.087 0.0069 1.1757 16.4711 97.36 ± 0.04 3.732 ± 0.023
bartowski Q6_K 22376 22148 0.04659 ± 0.00156 2.447 ± 0.079 0.0054 0.3869 12.4783 97.58 ± 0.03 3.756 ± 0.023
bartowski Q6_K_L 22963 22736 0.04494 ± 0.00151 2.301 ± 0.079 0.0046 0.3704 11.9331 97.87 ± 0.03 3.768 ± 0.023
NikiKrutan NIKI-Q6_K_L 23000 22785 0.03539 ± 0.00132 2.125 ± 0.079 0.0037 0.2434 10.4939 98.06 ± 0.03 3.782 ± 0.024
unsloth 3.0 UD-Q6_K_L 23073 22738 0.06866 ± 0.00198 2.797 ± 0.085 0.0055 0.8841 15.8937 97.66 ± 0.03 3.753 ± 0.023
AtomicChat AD-Q6_K 23847 23493 0.04172 ± 0.00146 2.283 ± 0.082 0.0040 0.3349 11.6762 98.00 ± 0.03 3.785 ± 0.024
unsloth 3.0 UD-Q6_K_XL 24127 23792 0.06625 ± 0.00196 2.756 ± 0.087 0.0046 0.7587 15.7494 97.78 ± 0.03 3.758 ± 0.023
unsloth 3.0 preview UD-Q6_K_XL 24723 24334 0.03966 ± 0.00145 2.136 ± 0.081 0.0031 0.2803 11.7220 98.14 ± 0.03 3.768 ± 0.023
### independent-v2 corpus
Warning! Huge table inside! *Lower is better **except same top p***

Qwen3.8-27B — independent-v2 corpus

Source Label Size, MiB* Size w/o MTP, MiB Mean KLD RMS Δp KLD p95 KLD p99 KLD p99.9 Same top p, % PPL
unsloth 3.0 UD-IQ1_S 6240 * 5905 0.60813 ± 0.00186 23.522 ± 0.069 2.1155 4.1158 7.1822 66.99 ± 0.11 12.776 ± 0.078
unsloth 3.0 UD-IQ1_M 6752 * 6417 0.42429 ± 0.00145 19.376 ± 0.065 1.4844 3.2142 6.3670 72.03 ± 0.10 10.723 ± 0.064
unsloth 3.0 UD-IQ2_XXS 7264 * 6929 0.31338 ± 0.00120 16.361 ± 0.062 1.1123 2.5462 5.5458 75.58 ± 0.10 9.676 ± 0.058
AtomicChat AD-IQ1_M 8104 7826 0.28422 ± 0.00107 15.475 ± 0.060 0.9671 2.2719 5.1406 76.70 ± 0.10 9.390 ± 0.055
unsloth 3.0 UD-IQ2_S 8319 * 7984 0.17177 ± 0.00080 11.770 ± 0.057 0.6043 1.5694 4.2142 81.83 ± 0.09 8.561 ± 0.051
NikiKrutan NIKI-IQ2_XXS 8500 8316 0.18872 ± 0.00083 12.664 ± 0.058 0.6395 1.6962 4.4760 80.75 ± 0.09 8.639 ± 0.050
AtomicChat AD-IQ2_XXS 8561 8283 0.21409 ± 0.00088 13.325 ± 0.057 0.7081 1.8009 4.7080 79.50 ± 0.09 8.848 ± 0.052
unsloth 3.0 preview UD-IQ2_XXS 8593 8382 0.17763 ± 0.00079 12.199 ± 0.058 0.5790 1.5806 4.3460 80.77 ± 0.09 8.656 ± 0.051
bartowski IQ2_XXS 8958 8730 0.26214 ± 0.00105 15.046 ± 0.062 0.8957 2.2519 5.2075 77.56 ± 0.09 9.283 ± 0.055
NikiKrutan NIKI-IQ2_S 9100 8925 0.13775 ± 0.00068 10.720 ± 0.056 0.4627 1.2822 3.8480 83.43 ± 0.08 8.307 ± 0.049
unsloth 3.0 UD-Q2_K_XL 9374 9039 0.10854 ± 0.00059 9.232 ± 0.053 0.3701 1.0421 3.4345 85.33 ± 0.08 8.142 ± 0.048
AtomicChat AD-IQ2_XS 9432 9153 0.14253 ± 0.00069 10.673 ± 0.053 0.4686 1.2057 3.6325 83.14 ± 0.08 8.336 ± 0.049
bartowski IQ2_XS 9524 9296 0.19347 ± 0.00085 12.798 ± 0.058 0.6536 1.7233 4.5241 80.57 ± 0.09 8.736 ± 0.051
AtomicChat AD-IQ2_S-IQ2_XS 9748 9469 0.11888 ± 0.00059 9.742 ± 0.052 0.3880 1.0273 3.2534 84.46 ± 0.08 8.171 ± 0.048
NikiKrutan NIKI-IQ3_XXS 9800 9622 0.10149 ± 0.00053 9.141 ± 0.052 0.3371 0.9496 3.0609 85.73 ± 0.08 8.054 ± 0.047
bartowski IQ2_S 9818 9590 0.17692 ± 0.00081 12.181 ± 0.058 0.5928 1.5965 4.4122 81.30 ± 0.09 8.616 ± 0.051
unsloth 3.0 preview UD-IQ2_M 9842 9636 0.10846 ± 0.00054 9.425 ± 0.052 0.3404 0.9451 3.1076 84.34 ± 0.08 8.168 ± 0.048
unsloth 3.0 preview UD-Q2_K_XL 10182 9976 0.09559 ± 0.00049 8.815 ± 0.050 0.2962 0.8225 2.8128 85.05 ± 0.08 8.074 ± 0.048
bartowski IQ2_M 10370 10142 0.12842 ± 0.00061 10.298 ± 0.054 0.4225 1.1543 3.5092 83.75 ± 0.08 8.262 ± 0.048
unsloth 3.0 UD-IQ3_XXS 10428 10094 0.07049 ± 0.00044 7.453 ± 0.050 0.2356 0.6935 2.3984 87.90 ± 0.07 7.894 ± 0.046
NikiKrutan NIKI-IQ3_XXS 10500 10322 0.07886 ± 0.00043 8.066 ± 0.047 0.2452 0.6858 2.4708 87.33 ± 0.07 7.895 ± 0.045
AtomicChat AD-IQ2_S 10626 10347 0.08589 ± 0.00046 8.298 ± 0.047 0.2786 0.7362 2.4607 86.99 ± 0.08 7.958 ± 0.046
mudler APEX-I-Nano 10720 10289 0.09942 ± 0.00053 8.882 ± 0.053 0.3395 0.9385 3.0039 86.06 ± 0.08 8.091 ± 0.048
byteshape IQ3_S-3.44bpw 11226 11058 0.06952 ± 0.00044 7.384 ± 0.051 0.2318 0.7017 2.6157 88.54 ± 0.07 7.894 ± 0.046
NikiKrutan NIKI-IQ3_S 11300 11116 0.05702 ± 0.00035 6.788 ± 0.046 0.1825 0.5295 1.9456 89.37 ± 0.07 7.783 ± 0.045
unsloth 3.0 preview UD-IQ3_XXS 11362 11167 0.05629 ± 0.00037 6.829 ± 0.049 0.1801 0.5481 2.1643 89.45 ± 0.07 7.837 ± 0.046
unsloth 3.0 UD-IQ3_S 11483 11148 0.04598 ± 0.00031 5.902 ± 0.044 0.1538 0.4554 1.6954 90.34 ± 0.07 7.735 ± 0.045
AtomicChat AD-IQ3_XXS 11516 11238 0.05924 ± 0.00034 6.850 ± 0.043 0.1870 0.5114 1.8070 89.11 ± 0.07 7.795 ± 0.045
byteshape IQ4_XS-3.67bpw 11975 11806 0.05568 ± 0.00038 6.634 ± 0.050 0.1842 0.5734 2.1963 89.73 ± 0.07 7.804 ± 0.046
bartowski IQ3_XXS 12042 11814 0.06745 ± 0.00040 7.340 ± 0.048 0.2191 0.6406 2.3279 88.23 ± 0.07 7.862 ± 0.046
NikiKrutan NIKI-IQ3_S 12100 11890 0.04349 ± 0.00031 5.962 ± 0.047 0.1365 0.4072 1.7680 90.25 ± 0.07 7.746 ± 0.045
AtomicChat AD-IQ3_S-IQ3_XXS 12383 12104 0.03803 ± 0.00026 5.393 ± 0.040 0.1243 0.3376 1.3080 90.88 ± 0.07 7.687 ± 0.045
unsloth 3.0 UD-Q3_K_XL 12537 12203 0.03045 ± 0.00023 4.843 ± 0.041 0.1006 0.2859 1.1419 92.00 ± 0.06 7.657 ± 0.045
bartowski IQ3_XS 12713 12485 0.05041 ± 0.00031 6.326 ± 0.042 0.1600 0.4567 1.6053 89.92 ± 0.07 7.747 ± 0.045
unsloth 3.0 preview UD-Q3_K_XL 12818 12620 0.03249 ± 0.00024 5.156 ± 0.041 0.1015 0.2971 1.2002 91.63 ± 0.06 7.687 ± 0.045
NikiKrutan NIKI-IQ4_XS 13000 12790 0.03034 ± 0.00025 4.923 ± 0.042 0.0942 0.2761 1.3123 91.81 ± 0.06 7.663 ± 0.045
byteshape IQ4_XS-4.00bpw 13030 12835 0.03808 ± 0.00028 5.450 ± 0.045 0.1261 0.4026 1.5728 91.46 ± 0.06 7.707 ± 0.045
AtomicChat AD-IQ3_S 13197 12971 0.02895 ± 0.00021 4.662 ± 0.036 0.0957 0.2545 0.9263 92.02 ± 0.06 7.666 ± 0.045
bartowski IQ3_M 13259 13032 0.04607 ± 0.00028 6.120 ± 0.041 0.1435 0.4139 1.5318 90.53 ± 0.07 7.750 ± 0.045
mudler APEX-I-Mini 13297 12867 0.04092 ± 0.00030 5.675 ± 0.045 0.1354 0.4037 1.5444 91.03 ± 0.06 7.725 ± 0.046
unsloth 3.0 UD-IQ4_XS 13593 13258 0.02075 ± 0.00018 4.029 ± 0.040 0.0662 0.1947 0.9067 93.34 ± 0.06 7.605 ± 0.044
AtomicChat AD-IQ4_XS-IQ3_S 13769 13534 0.02314 ± 0.00017 4.180 ± 0.036 0.0744 0.2036 0.8089 92.82 ± 0.06 7.635 ± 0.045
NikiKrutan NIKI-IQ4_XS 14000 13790 0.01777 ± 0.00016 3.754 ± 0.038 0.0552 0.1621 0.6999 93.75 ± 0.06 7.600 ± 0.045
byteshape IQ4_XS-4.40bpw 14344 14115 0.01946 ± 0.00017 3.825 ± 0.034 0.0639 0.1949 0.7830 93.69 ± 0.06 7.588 ± 0.044
unsloth 3.0 UD-Q4_K_S 14647 14312 0.01431 ± 0.00015 3.376 ± 0.038 0.0453 0.1370 0.6510 94.58 ± 0.05 7.574 ± 0.044
bartowski IQ4_XS 14847 14619 0.01530 ± 0.00015 3.445 ± 0.036 0.0480 0.1374 0.5749 94.27 ± 0.05 7.588 ± 0.045
unsloth 3.0 preview IQ4_XS 14978 14725 0.01467 ± 0.00015 3.339 ± 0.034 0.0466 0.1319 0.5580 94.33 ± 0.05 7.584 ± 0.044
NikiKrutan NIKI-IQ4_XS 15000 14785 0.01448 ± 0.00015 3.383 ± 0.035 0.0453 0.1371 0.6066 94.38 ± 0.05 7.578 ± 0.044
byteshape Q5_K_S-4.72bpw 15394 15144 0.01562 ± 0.00015 3.447 ± 0.036 0.0505 0.1536 0.6493 94.31 ± 0.05 7.571 ± 0.044
unsloth 3.0 UD-Q4_K_M 15702 15367 0.01037 ± 0.00013 2.832 ± 0.032 0.0331 0.0952 0.4359 95.33 ± 0.05 7.558 ± 0.044
AtomicChat AD-IQ4_XS 15748 15490 0.01102 ± 0.00013 2.886 ± 0.032 0.0349 0.0989 0.4484 95.10 ± 0.05 7.556 ± 0.044
bartowski Q4_K_S 15939 15711 0.01323 ± 0.00013 3.163 ± 0.031 0.0424 0.1194 0.4670 94.50 ± 0.05 7.571 ± 0.044
lmstudio-community Q4_K_M 16032 15781 0.01954 ± 0.00017 3.796 ± 0.034 0.0631 0.1724 0.6741 93.41 ± 0.06 7.601 ± 0.045
NikiKrutan NIKI-IQ4_XS 16100 15885 0.01211 ± 0.00014 3.070 ± 0.032 0.0373 0.1066 0.4506 94.69 ± 0.05 7.564 ± 0.044
unsloth 3.0 preview Q4_K_M 16314 16038 0.01211 ± 0.00013 3.017 ± 0.030 0.0384 0.1088 0.4570 94.75 ± 0.05 7.570 ± 0.044
AtomicChat AD-Q4_K 16328 16062 0.01002 ± 0.00013 2.769 ± 0.031 0.0319 0.0876 0.3724 95.26 ± 0.05 7.555 ± 0.044
unsloth 3.0 UD-Q4_K_XL 16746 16411 0.00768 ± 0.00012 2.410 ± 0.029 0.0242 0.0705 0.3030 95.88 ± 0.04 7.546 ± 0.044
bartowski Q4_K_M 16949 16721 0.01058 ± 0.00012 2.864 ± 0.032 0.0332 0.0953 0.4210 95.09 ± 0.05 7.563 ± 0.044
unsloth 3.0 preview UD-Q4_K_XL 17093 16821 0.00822 ± 0.00012 2.507 ± 0.030 0.0255 0.0754 0.3284 95.74 ± 0.05 7.548 ± 0.044
NikiKrutan NIKI-Q5_K_S 17300 17085 0.00745 ± 0.00012 2.396 ± 0.029 0.0232 0.0687 0.2908 95.96 ± 0.04 7.539 ± 0.044
AtomicChat AD-Q5_K-Q4_K 17688 17411 0.00687 ± 0.00012 2.297 ± 0.031 0.0216 0.0593 0.2554 96.01 ± 0.04 7.540 ± 0.044
unsloth 3.0 UD-Q5_K_S 17801 17466 0.00569 ± 0.00010 2.102 ± 0.028 0.0179 0.0520 0.2294 96.40 ± 0.04 7.539 ± 0.044
bartowski Q4_K_L 17849 17621 0.00997 ± 0.00013 2.753 ± 0.030 0.0319 0.0903 0.3688 95.34 ± 0.05 7.560 ± 0.044
byteshape Q5_K_M-5.60bpw 18249 17965 0.00675 ± 0.00011 2.248 ± 0.030 0.0211 0.0619 0.2567 96.12 ± 0.04 7.540 ± 0.044
unsloth 3.0 preview Q5_K_S 18377 18080 0.00575 ± 0.00009 2.109 ± 0.026 0.0178 0.0494 0.2115 96.25 ± 0.04 7.534 ± 0.044
NikiKrutan NIKI-Q5_K_M 18600 18385 0.00468 ± 0.00010 1.879 ± 0.025 0.0144 0.0422 0.1810 96.73 ± 0.04 7.527 ± 0.044
bartowski Q5_K_S 18769 18541 0.00527 ± 0.00010 1.982 ± 0.026 0.0163 0.0438 0.1773 96.41 ± 0.04 7.533 ± 0.044
unsloth 3.0 UD-Q5_K_M 18856 18521 0.00427 ± 0.00010 1.822 ± 0.030 0.0129 0.0366 0.1589 96.78 ± 0.04 7.527 ± 0.044
unsloth 3.0 preview Q5_K_M 18915 18606 0.00492 ± 0.00009 1.921 ± 0.027 0.0151 0.0402 0.1639 96.48 ± 0.04 7.530 ± 0.044
unsloth 3.0 preview UD-Q5_K_XL 19282 18958 0.00415 ± 0.00009 1.766 ± 0.026 0.0126 0.0353 0.1452 96.80 ± 0.04 7.530 ± 0.044
AtomicChat AD-Q5_K 19295 18990 0.00391 ± 0.00006 1.747 ± 0.026 0.0118 0.0331 0.1490 96.88 ± 0.04 7.526 ± 0.044
bartowski Q5_K_M 19791 19563 0.00439 ± 0.00011 1.837 ± 0.031 0.0133 0.0356 0.1430 96.68 ± 0.04 7.529 ± 0.044
unsloth 3.0 UD-Q5_K_XL 19910 19575 0.00322 ± 0.00010 1.550 ± 0.025 0.0101 0.0278 0.1239 97.32 ± 0.04 7.525 ± 0.044
NikiKrutan NIKI-Q6_K 20000 19785 0.00280 ± 0.00005 1.462 ± 0.021 0.0088 0.0250 0.1108 97.51 ± 0.04 7.519 ± 0.044
bartowski Q5_K_L 20540 20312 0.00402 ± 0.00011 1.738 ± 0.029 0.0125 0.0337 0.1400 96.97 ± 0.04 7.527 ± 0.044
unsloth 3.0 UD-Q6_K 20965 20631 0.00244 ± 0.00010 1.353 ± 0.029 0.0074 0.0207 0.0885 97.65 ± 0.03 7.521 ± 0.044
lmstudio-community Q6_K 21392 21060 0.00267 ± 0.00003 1.419 ± 0.020 0.0081 0.0210 0.0851 97.29 ± 0.04 7.521 ± 0.044
NikiKrutan NIKI-Q6_K 21400 21185 0.00192 ± 0.00008 1.192 ± 0.019 0.0056 0.0162 0.0714 97.93 ± 0.03 7.519 ± 0.044
unsloth 3.0 preview Q6_K 21824 21480 0.00209 ± 0.00010 1.224 ± 0.019 0.0059 0.0153 0.0603 97.58 ± 0.04 7.522 ± 0.044
AtomicChat AD-Q6_K-Q5_K 22012 21689 0.00248 ± 0.00004 1.404 ± 0.030 0.0077 0.0217 0.0945 97.67 ± 0.03 7.521 ± 0.044
unsloth 3.0 UD-Q6_K_M 22019 21684 0.00198 ± 0.00010 1.210 ± 0.030 0.0059 0.0168 0.0710 97.99 ± 0.03 7.519 ± 0.044
bartowski Q6_K 22376 22148 0.00176 ± 0.00009 1.136 ± 0.016 0.0050 0.0129 0.0506 97.69 ± 0.03 7.517 ± 0.044
bartowski Q6_K_L 22963 22736 0.00142 ± 0.00007 1.001 ± 0.015 0.0042 0.0117 0.0454 98.19 ± 0.03 7.516 ± 0.044
NikiKrutan NIKI-Q6_K_L 23000 22785 0.00124 ± 0.00002 0.986 ± 0.024 0.0038 0.0106 0.0428 98.23 ± 0.03 7.515 ± 0.044
unsloth 3.0 UD-Q6_K_L 23073 22738 0.00163 ± 0.00010 1.113 ± 0.035 0.0046 0.0133 0.0574 98.18 ± 0.03 7.517 ± 0.044
AtomicChat AD-Q6_K 23847 23493 0.00105 ± 0.00002 0.889 ± 0.021 0.0032 0.0087 0.0344 98.45 ± 0.03 7.517 ± 0.044
unsloth 3.0 UD-Q6_K_XL 24127 23792 0.00139 ± 0.00009 1.029 ± 0.034 0.0039 0.0111 0.0504 98.34 ± 0.03 7.517 ± 0.044
unsloth 3.0 preview UD-Q6_K_XL 24723 24334 0.00102 ± 0.00007 0.851 ± 0.011 0.0029 0.0083 0.0359 98.41 ± 0.03 7.515 ± 0.044
## MTP
I've tested MTP performance and discovered that there is no point to keep MTP head higher than `IQ4_XS`. `Q3_K` is near but slightly worse (used for lower quants). Exact allocation of IQ4_XS/Q3_K is chosen by allocator (not pinned). Lower than that there is considerable quality drop, so not used in my quants. Higher is just waste of model size and VRAM. For numbers check my previous [GRM-2.6-Plus-0628](https://huggingface.co/NikiKrutan/GRM-2.6-Plus-0628-MTP-GGUF) quant's MTP section. I've done fast tests on other models (same architecture). Same conclusions. Also I've additionally checked fast if there is any degradation on `IQ4_XS` vs `Q8_0` with longer context (tested up to 28K), none found. And temperature = 0.6, still no evidence of degradation. But I must say that doing this test properly and scientifically much time and effort is needed. Since not 0 temperature gives different results: many runs needed to stabilize. Reasonably long context (100K+ from my opinion) is very slow on my hardware. So there actually may be some degradation. But that should be proven and not taken by "it seems" or "everybody knows". *If there is such comprehensive analysis already, please let me know.* Basically I use these parameters for MTP: `--spec-draft-n-max` 5 `--spec-draft-p-min` 0.8. Primary reason of poor results with MTP is ignoring `--spec-draft-p-min`. Don't put obvious crap in your drafts. But let drafter do more if it is sure enough. MTP performance speed-up is highly dependable on hardware, configuration and specific context. So to decide what is best for your case you should test yourself. Also don't use other `llama.cpp` drafters like `ngram-mod` with MTP. MTP wastes time even if `ngram-mod` already done draft. That is how `llama.cpp` is programmed. Not obvious. I have created experimental fork to overcome this issue (and added much better `ngram-mod-v2` + minor fixes): https://github.com/NikiKrutan/niki-llama.cpp. I use it myself on a daily basis, but it is more like crude draft than real working fork. It breaks some llama.cpp behavior. But it gives another ~1.5x speed-up on top of MTP for my use cases.
## Chat template Chat template included with my gguf's is v22.2 from [froggeric](https://huggingface.co/froggeric/Qwen-Fixed-Chat-Templates). There may be newer version as it is often updated. If you have any problems with it you may also try one from [Moore2877](https://huggingface.co/Moore2877/Qwen-Fixed-Chat-Templates-llamacpp). ## Multimodality I've quantized mmproj following [DAXZEIT's receipt for Qwen3.6](https://huggingface.co/DAXZEIT/Qwen3.6-27B-mmproj-hybrid-Q8_0-F16-gguf) that I used in Qwen3.6 era. It reduces mmproj size saving practically same quality. Why reducing size? Well, it is probably not needed for most use cases. But with this I can put my 17300 quant with q8_0 K-cache and q4_0 V-cache + MTP with the same KV-cache quantization + mmproj + 262K full context in my 2x16Gb VRAM. With BF16 mmproj I can't do it despite very little difference in size. Every MiB matters when you are about quality. Also any standard Qwen3.8-27B mmproj should work if you want full BF16 one. ## Acknowledgments - [Qwen](https://huggingface.co/Qwen/Qwen3.8-27B) - great base model. Really great! - [llama.cpp](https://github.com/ggml-org/llama.cpp) - quantization and evaluation framework. - [unsloth](https://huggingface.co/unsloth/Qwen3.8-27B-GGUF) - quants for comparison. - [AtomicChat](https://huggingface.co/AtomicChat/Qwen3.8-27B-GGUF) - quants for comparison. - [bartowski](https://huggingface.co/bartowski/Qwen3.8-27B-GGUF) - quants for comparison. - [lmstudio-community](https://huggingface.co/lmstudio-community/Qwen3.8-27B-GGUF) - quants for comparison. - [mudler](https://huggingface.co/mudler/Qwen3.8-27B-APEX-GGUF) - quants for comparison. - [byteshape](https://huggingface.co/byteshape/Qwen3.8-27B-GGUF) - quants for comparison. - [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) - data for evaluation dataset. - [caselaw_access_project](https://huggingface.co/datasets/common-pile/caselaw_access_project) - data for evaluation dataset. - [open-web-math](https://huggingface.co/datasets/open-web-math/open-web-math) - data for evaluation dataset. - [recursal](https://huggingface.co/datasets/recursal/arXiv-CC0-v0.5) - data for evaluation dataset. - [DAXZEIT](https://huggingface.co/DAXZEIT/Qwen3.6-27B-mmproj-hybrid-Q8_0-F16-gguf) - mmproj receipt. - [michaelw9999](https://github.com/michaelw9999/advanced-gguf-quantizer) - some ideas to start with. - Required attributions for exact science papers used for evaluation dataset are on [dataset page](https://huggingface.co/datasets/NikiKrutan/independent).