Instructions to use NikiKrutan/Qwen3.8-27B-MTP-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use NikiKrutan/Qwen3.8-27B-MTP-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS
# Run inference directly in the terminal:
llama cli -hf NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS
# Run inference directly in the terminal:
llama cli -hf NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS
# Run inference directly in the terminal:
./llama-cli -hf NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS

Use Docker

docker model run hf.co/NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS

LM Studio
Jan

vLLM

How to use NikiKrutan/Qwen3.8-27B-MTP-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "NikiKrutan/Qwen3.8-27B-MTP-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "NikiKrutan/Qwen3.8-27B-MTP-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS

Ollama
How to use NikiKrutan/Qwen3.8-27B-MTP-GGUF with Ollama:
```
ollama run hf.co/NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS
```
Unsloth Desktop

How to use NikiKrutan/Qwen3.8-27B-MTP-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use NikiKrutan/Qwen3.8-27B-MTP-GGUF with Docker Model Runner:
```
docker model run hf.co/NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS
```

Lemonade

How to use NikiKrutan/Qwen3.8-27B-MTP-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS

Run and chat with the model

lemonade run user.Qwen3.8-27B-MTP-GGUF-IQ3_XXS

List all available models

lemonade list

Hermes Agent

How to use NikiKrutan/Qwen3.8-27B-MTP-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use NikiKrutan/Qwen3.8-27B-MTP-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "NikiKrutan/Qwen3.8-27B-MTP-GGUF:IQ3_XXS" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Quality-first GGUF quants of Qwen3.8-27B

Consider this fine-tune instead, it is even better: Kiwen1.1-27B

In the process of creating these quants:

95 hours of 2xGPU near 100% load were spent!
27 TB of not so good ggufs were materialized, measured and put to trash (all were usable)!
72 man-hours were spent!

These quants were produced by my quality-first experimental tool, niki-allocator. The main difference of my method is per-tensor type allocation (not per tensor groups, layers, community lore etc.) Each tensor gets what is estimated to be best for it. After that best allocations are tested with real KLD runs to choose the winner.

Source model

Quantization quality comparison

To prove my method I've performed comparison with quants from other providers:

unsloth, AtomicChat, bartowski, lmstudio-community, mudler, byteshape

Lower is better

Main metrics

Tail metrics

p99.9 are extremely corpus-driven with usual test corpus sizes of ~300-400K tokens

Same top probability

Useful for temperature 0, otherwise RMS Δp should be considered.

Note that graphs show 1 - same top p, so lower is better.

Perplexity (PPL)

Very crude noisy metric, cheaper to compute though, that's why so popular

Notes

TL;DR difference of corpora for end user: niki-small-plus-2 - coding, agentic, reasoning, hard logic prompts, English, Russian; independent-v2 - multilingual, English, web texts, science papers, math, legal, no reasoning or agentic traces, no code.

Unsloth 3.0 preview, AtomicChat, bartowski, lmstudio-community quants for comparison were downloaded on August 16, 2026. Unsloth 3.0, mudler, byteshape quants for comparison were downloaded on August 21, 2026. I've deliberately dropped some known bad quants (legacy quant types and such) to save my time on downloading and measuring.

Unsloth v3.0 was introduced after I finished my work. I added them for clean honest comparison. I don't like their marketing attitude and evidence presentation (just drop in their best graph without details on how exactly it was measured and if it is meaningful metric for end user). But after all they provided really good quants (mostly).

Unsloth v3.0 dropped MTP for lower quants reasoning that they are not needed. Well, I can't agree. From my findings MTP works well on lower quants. I added these no-MTP quants to comparison, but added fixed-size correction of their MTP v3.0 fixed-choice for all their non-MTP quants. Yeah, I know.. So for "honest" scientific comparison of only target models without strange choices of MTP quantization, here is comparison for all quants where only size of target model is accounted. This is not that honest for the end user though. End user downloads quant and either fits it in his VRAM/RAM or not, poor choice of oversized MTP can't be excuse here. Quant provider should provide MTP quantization based on their research, not just drop in something, or just delete it. It is part of a quant. But here you go:

All graphs with MTP size subtracted

Main metrics

Tail metrics

p99.9 are extremely corpus-driven with usual test corpus sizes of ~300-400K tokens

Same top probability

Useful for temperature 0, otherwise RMS Δp should be considered.

Note that graphs show 1 - same top p, so lower is better.

Perplexity (PPL)

Very crude noisy metric, cheaper to compute though, that's why so popular

See the tables below for exact numbers for all quants and metrics

Sizes

I've chosen 15 sizes to fill 8500-23000 MiB with geometric progression and rounded to 100 MiB

niki-allocator is able to produce allocation with ~1 MiB precision.

Which one to choose?

If you are after quality: the bigger - the better. So it is always a compromise of what you can afford considering context size and other constraints. Choose the biggest you can fit.

About speed

I haven't measured speed since it is highly hardware and configuration dependable. What is faster for me may be slower for you. If the speed is the only that matters and you have Blackwell - get some NVFP4 quant (but quality is considerably worse for the same quant size). Otherwise you should measure speed yourself on your hardware and your configuration to make real decision. By the way, on my setup ubatch is the parameter that has huge impact on speed. On 27B models 384 was found to be optimal in my case (2x5060Ti), 512 or 256 are much slower. So probably you should test this first.

Quant names

Hugging Face requires quant type in the filename to properly show it. But all my quants are mixed by design. I decided to include dominating quant type (by size, not by tensor count) to the name therefore. It may easily be less than 50% of the model though. So choose by size, ignore listed quant type. For my quants: bigger --> better quality. Always.

Methodology

niki-allocator

What is niki-allocator

It is quality-first per-tensor quantization allocator for llama.cpp llama-quantize.

Allocator decides, per tensor, which quant type to use so that the resulting GGUF hits an exact target file size with the best estimated quality.

It is not direct measurement of KLD-metrics, it is only estimation. But as you can see from real KLD evaluations estimation is good enough.

Raw results are already good. But I added some calibration levers on top to make it better (for Qwen3.5-0.8B I used uncalibrated estimation). These levers need series of KLD evaluations to move estimation closer to reality, so it takes hours-days of GPU time on my consumer setup.

On top of this I added bayesian search seeded with best results from levers to move calibration even further. This is very GPU-intensive, literally took days (nights actually) of GPU time on my setup.

Corpora

Corpora choice, differences, details

imatrix: niki-small-2 corpus - ~330K tokens from my own agentic sessions (English, Russian, design docs with hard logic, code, reasoning). Deduped, cleaned, filtered by simhash similarity to include only really different turns. Turn-aligned by tokenizer so that each chunk contains exactly 1 turn. Short turns dropped. Used for imatrix only.
Calibration of niki-allocator: hardest 16 chunks of niki-small-plus-2 for faster crude KLD calibration (full corpus is too slow on my hardware).
Evaluation 1: niki-small-plus-2 corpus - ~400K tokens from my own agentic sessions (English, Russian, design docs with hard logic, code, reasoning). Deduped, cleaned, filtered by simhash similarity to include only really different turns. Turn-aligned by tokenizer so that each chunk contains exactly 1 turn. Short turns dropped. Does not include any exact or similar (by simhash) agentic turns from niki-small. So while it is from the same domain as imatrix-corpus, it is different.
Evaluation 2 (truly independent this time): independent-v2 corpus - ~390 K tokens. This corpus is random compilation of top-13 languages prose (HuggingFaceFW/fineweb-2) excluding English (it is everywhere in other domains already), legal domain (common-pile/caselaw_access_project), math domain (open-web-math/open-web-math), science domain (recursal/arXiv-CC0-v0.5). Code and agentic data wasn't included since my quants are highly tuned for these already (it wouldn't be honest comparison). This dataset wasn't seen during calibration or imatrix at all. Used only for final evaluation before publishing. So my quants are not anyhow tuned for this dataset. Foreign quants haven't seen this exact dataset as well (random pick).

Notes:

independent-v2 corpus will be published here. Note that you can't redistribute it without proper attributions due to license of some included papers.
niki-* corpora are not published. There is nothing really special actually. They just contain my private information. Sorry for inconvenience.
Why not wikitext-2 test? While wiki may be considered as independent evaluation corpus for my quants. It easily may not be true for others quants (wikitext-2 is the most popular corpus somehow for such jobs). They may have seen it (or part of it) during imatrix creation or their methodology calibration. So for really independent evaluation there should be corpus that wasn't seen anyhow by any quants and even better if it wasn't seen during base model creation. Also wiki is mostly homogenous (not very easy, but there are very few hard tokens and chunks). My corpus is everything from "walk in the park" to "high mountain climbing". Its tail metrics are much higher. It is actually more representative for the real model work (if crawling wiki is not your main use case of course). So I use my own corpus because I need these quants to be good in my tasks. And independent-v2 as truly independent evaluation.

Chunk sizes

imatrix (niki-small-2): 8064 tokens (384 x 21). 41 chunks total. To catch reasoning and long turn traces, not just basic chat.
niki-small-plus-2 evaluation: 6144 tokens (384 x 16). 65 chunks total. To catch reasoning and long turns, but to keep chunks amount somewhat reasonable (that's why lower than for imatrix).
independent-v2 evaluation: 2304 tokens (384 x 6). 171 chunks total. Long enough for chat and basic tasks.

Why 384 magic number?

My hardware is fastest with ubatch 384, so I use it.

Metrics

All metrics were evaluated with llama-perplexity against Q8_0 standard quant (due to hardware constraints).

Quantization was performed with patched llama-quantize from BF_16 source. Fully compatible with original llama.cpp.

All my and foreign quants were evaluated against the same base.

llama.cpp commit 876a4321163249c43ca4e986818fab5ab081f282.

The main metrics during calibration of niki-allocator were mean KLD and RMS Δp.

Better PPL doesn't always mean better other more important metrics. So it can't be used as a main metric (some people still use). It is just fast indicator (no need for logits base) to check for big problems. It can't be main quality metric. Don't make or accept comparisons done only by PPL. It is noise.

Notes on p99 KLD, p99.9 KLD, max KLD.

With such a small corpora these metrics are not very robust. While p99 KLD can give some plausible comparison for the allocation method as a whole, p99.9 is extremely corpus-driven (too little data), max KLD is just random at any corpus size (one hardest token of the whole dataset, saying nothing about other hard tokens). So I don't tune to these. And anybody who does with less than several M tokens corpus is just over-fitting to their exact corpus. Please correct me if I am wrong.

Evaluation results

niki-small-plus-2 corpus

Warning! Huge table inside!

Lower is better except same top p

Qwen3.8-27B — niki-small-plus-2 corpus

Source	Label	Size, MiB*	Size w/o MTP, MiB	Mean KLD	RMS Δp	KLD p95	KLD p99	KLD p99.9	Same top p, %	PPL
unsloth 3.0	UD-IQ1_S	6240 *	5905	0.59681 ± 0.00410	21.210 ± 0.063	1.6709	10.2128	22.0603	75.44 ± 0.10	4.610 ± 0.026
unsloth 3.0	UD-IQ1_M	6752 *	6417	0.53205 ± 0.00457	18.591 ± 0.066	1.2365	12.3683	23.5830	78.62 ± 0.09	4.263 ± 0.025
unsloth 3.0	UD-IQ2_XXS	7264 *	6929	0.42809 ± 0.00430	15.342 ± 0.064	0.8708	11.1274	23.3418	81.79 ± 0.09	4.350 ± 0.028
AtomicChat	AD-IQ1_M	8104	7826	0.40475 ± 0.00409	15.342 ± 0.064	0.8258	10.2505	23.0421	81.78 ± 0.09	3.905 ± 0.022
unsloth 3.0	UD-IQ2_S	8319 *	7984	0.35110 ± 0.00429	12.211 ± 0.069	0.5158	11.1857	23.2209	85.77 ± 0.08	4.331 ± 0.029
NikiKrutan	NIKI-IQ2_XXS	8500	8316	0.29969 ± 0.00364	11.724 ± 0.062	0.4843	8.4590	21.2619	85.49 ± 0.08	4.065 ± 0.025
AtomicChat	AD-IQ2_XXS	8561	8283	0.40775 ± 0.00449	13.996 ± 0.067	0.6582	12.1255	23.5798	83.37 ± 0.08	4.615 ± 0.032
unsloth 3.0 preview	UD-IQ2_XXS	8593	8382	0.29882 ± 0.00369	11.384 ± 0.059	0.4793	8.3221	21.7652	85.56 ± 0.08	4.171 ± 0.027
bartowski	IQ2_XXS	8958	8730	0.40331 ± 0.00404	14.430 ± 0.059	0.8150	10.3693	22.2155	81.90 ± 0.09	4.419 ± 0.029
NikiKrutan	NIKI-IQ2_S	9100	8925	0.29060 ± 0.00385	10.310 ± 0.063	0.3783	9.0926	22.4354	87.09 ± 0.07	4.178 ± 0.028
unsloth 3.0	UD-Q2_K_XL	9374	9039	0.23685 ± 0.00343	9.208 ± 0.063	0.2930	6.8906	20.9554	88.65 ± 0.07	3.807 ± 0.023
AtomicChat	AD-IQ2_XS	9432	9153	0.33312 ± 0.00420	11.466 ± 0.066	0.4421	10.7630	22.8707	86.09 ± 0.08	4.395 ± 0.030
bartowski	IQ2_XS	9524	9296	0.38705 ± 0.00434	12.842 ± 0.062	0.6032	11.7030	22.9351	83.97 ± 0.08	4.470 ± 0.031
AtomicChat	AD-IQ2_S-IQ2_XS	9748	9469	0.29551 ± 0.00394	10.421 ± 0.067	0.3584	9.7478	22.0738	87.27 ± 0.07	4.248 ± 0.029
NikiKrutan	NIKI-IQ3_XXS	9800	9622	0.24855 ± 0.00365	8.648 ± 0.059	0.2652	8.1560	22.2199	88.86 ± 0.07	3.947 ± 0.025
bartowski	IQ2_S	9818	9590	0.37596 ± 0.00436	12.518 ± 0.066	0.5408	11.8180	22.9918	84.70 ± 0.08	4.483 ± 0.031
unsloth 3.0 preview	UD-IQ2_M	9842	9636	0.29107 ± 0.00401	9.528 ± 0.066	0.2816	10.4280	21.9791	87.91 ± 0.07	4.028 ± 0.026
unsloth 3.0 preview	UD-Q2_K_XL	10182	9976	0.23467 ± 0.00349	9.201 ± 0.067	0.2472	7.3177	21.1131	88.66 ± 0.07	4.010 ± 0.025
bartowski	IQ2_M	10370	10142	0.32379 ± 0.00419	10.543 ± 0.064	0.3837	10.7744	22.9631	86.66 ± 0.08	4.328 ± 0.030
unsloth 3.0	UD-IQ3_XXS	10428	10094	0.21417 ± 0.00346	7.863 ± 0.069	0.1830	7.0054	21.3873	90.48 ± 0.07	3.886 ± 0.024
NikiKrutan	NIKI-IQ3_XXS	10500	10322	0.19533 ± 0.00309	8.268 ± 0.062	0.2035	5.4233	19.7968	89.92 ± 0.07	3.712 ± 0.021
AtomicChat	AD-IQ2_S	10626	10347	0.24532 ± 0.00358	9.189 ± 0.066	0.2605	7.7219	21.6210	88.96 ± 0.07	4.076 ± 0.026
mudler	APEX-I-Nano	10720	10289	0.28538 ± 0.00385	10.279 ± 0.070	0.3606	9.3696	21.9993	87.91 ± 0.07	4.286 ± 0.029
byteshape	IQ3_S-3.44bpw	11226	11058	0.20084 ± 0.00331	7.754 ± 0.072	0.1755	6.1658	20.8096	91.00 ± 0.06	3.955 ± 0.025
NikiKrutan	NIKI-IQ3_S	11300	11116	0.18301 ± 0.00312	6.935 ± 0.065	0.1429	5.7343	20.1450	91.42 ± 0.06	3.845 ± 0.024
unsloth 3.0 preview	UD-IQ3_XXS	11362	11167	0.20210 ± 0.00335	7.402 ± 0.071	0.1454	6.8158	20.7976	91.19 ± 0.06	3.714 ± 0.022
unsloth 3.0	UD-IQ3_S	11483	11148	0.18327 ± 0.00331	6.523 ± 0.069	0.1247	5.5946	21.1550	92.14 ± 0.06	3.837 ± 0.024
AtomicChat	AD-IQ3_XXS	11516	11238	0.19840 ± 0.00325	7.755 ± 0.068	0.1677	6.1414	20.4022	90.75 ± 0.07	3.934 ± 0.025
byteshape	IQ4_XS-3.67bpw	11975	11806	0.17338 ± 0.00312	6.696 ± 0.072	0.1239	5.1036	20.5042	92.21 ± 0.06	3.728 ± 0.022
bartowski	IQ3_XXS	12042	11814	0.19534 ± 0.00327	8.055 ± 0.070	0.1887	5.6009	20.9745	90.41 ± 0.07	3.779 ± 0.023
NikiKrutan	NIKI-IQ3_S	12100	11890	0.17271 ± 0.00314	6.316 ± 0.069	0.1059	5.3612	20.4380	92.16 ± 0.06	3.740 ± 0.023
AtomicChat	AD-IQ3_S-IQ3_XXS	12383	12104	0.20045 ± 0.00341	6.686 ± 0.072	0.1206	7.2896	20.8107	91.87 ± 0.06	3.799 ± 0.023
unsloth 3.0	UD-Q3_K_XL	12537	12203	0.17021 ± 0.00325	6.254 ± 0.079	0.0846	5.2435	20.9727	93.10 ± 0.06	3.712 ± 0.023
bartowski	IQ3_XS	12713	12485	0.19216 ± 0.00319	7.144 ± 0.067	0.1438	6.3475	19.8683	91.29 ± 0.06	3.865 ± 0.024
unsloth 3.0 preview	UD-Q3_K_XL	12818	12620	0.14791 ± 0.00288	6.059 ± 0.073	0.0843	4.0237	19.7203	93.02 ± 0.06	3.698 ± 0.022
NikiKrutan	NIKI-IQ4_XS	13000	12790	0.14490 ± 0.00287	5.600 ± 0.071	0.0753	3.8287	19.8499	93.17 ± 0.06	3.757 ± 0.023
byteshape	IQ4_XS-4.00bpw	13030	12835	0.15545 ± 0.00301	5.622 ± 0.071	0.0842	4.3696	20.4840	93.33 ± 0.06	3.813 ± 0.024
AtomicChat	AD-IQ3_S	13197	12971	0.15641 ± 0.00298	5.882 ± 0.070	0.0908	4.4304	20.3319	92.90 ± 0.06	3.820 ± 0.024
bartowski	IQ3_M	13259	13032	0.17815 ± 0.00308	7.029 ± 0.069	0.1270	5.3719	19.9435	91.88 ± 0.06	3.880 ± 0.024
mudler	APEX-I-Mini	13297	12867	0.20655 ± 0.00349	7.053 ± 0.074	0.1328	7.2564	21.9309	91.81 ± 0.06	3.904 ± 0.025
unsloth 3.0	UD-IQ4_XS	13593	13258	0.14809 ± 0.00301	5.766 ± 0.085	0.0524	4.2925	20.2216	94.27 ± 0.05	3.622 ± 0.021
AtomicChat	AD-IQ4_XS-IQ3_S	13769	13534	0.14858 ± 0.00294	5.325 ± 0.071	0.0713	4.2618	19.8525	93.39 ± 0.06	3.834 ± 0.024
NikiKrutan	NIKI-IQ4_XS	14000	13790	0.12202 ± 0.00264	4.917 ± 0.075	0.0480	2.8676	18.7476	94.43 ± 0.05	3.696 ± 0.022
byteshape	IQ4_XS-4.40bpw	14344	14115	0.13487 ± 0.00284	4.788 ± 0.077	0.0465	3.6616	19.6340	94.69 ± 0.05	3.777 ± 0.023
unsloth 3.0	UD-Q4_K_S	14647	14312	0.12428 ± 0.00274	4.637 ± 0.079	0.0370	2.8792	19.6644	95.15 ± 0.05	3.688 ± 0.022
bartowski	IQ4_XS	14847	14619	0.12968 ± 0.00271	5.155 ± 0.081	0.0467	3.3971	18.9806	94.47 ± 0.05	3.905 ± 0.025
unsloth 3.0 preview	IQ4_XS	14978	14725	0.11673 ± 0.00258	4.766 ± 0.078	0.0423	2.7687	18.5722	94.75 ± 0.05	3.863 ± 0.024
NikiKrutan	NIKI-IQ4_XS	15000	14785	0.11079 ± 0.00253	4.354 ± 0.076	0.0362	2.3719	18.7210	95.06 ± 0.05	3.741 ± 0.023
byteshape	Q5_K_S-4.72bpw	15394	15144	0.13078 ± 0.00281	4.478 ± 0.080	0.0356	3.5353	19.4105	95.07 ± 0.05	3.763 ± 0.023
unsloth 3.0	UD-Q4_K_M	15702	15367	0.11679 ± 0.00264	4.138 ± 0.078	0.0300	2.6716	19.2746	95.47 ± 0.05	3.809 ± 0.024
AtomicChat	AD-IQ4_XS	15748	15490	0.11266 ± 0.00256	4.346 ± 0.078	0.0329	2.4330	18.9169	95.22 ± 0.05	3.769 ± 0.023
bartowski	Q4_K_S	15939	15711	0.12847 ± 0.00276	4.698 ± 0.079	0.0399	3.3534	19.5019	94.76 ± 0.05	3.763 ± 0.023
lmstudio-community	Q4_K_M	16032	15781	0.16945 ± 0.00319	6.184 ± 0.077	0.0850	5.4682	20.9170	93.02 ± 0.06	3.639 ± 0.021
NikiKrutan	NIKI-IQ4_XS	16100	15885	0.10334 ± 0.00243	4.209 ± 0.076	0.0313	2.0672	18.0062	95.27 ± 0.05	3.758 ± 0.023
unsloth 3.0 preview	Q4_K_M	16314	16038	0.12098 ± 0.00269	4.572 ± 0.079	0.0378	2.8831	19.4748	94.97 ± 0.05	3.769 ± 0.023
AtomicChat	AD-Q4_K	16328	16062	0.11616 ± 0.00265	4.432 ± 0.082	0.0300	2.6443	19.0882	95.42 ± 0.05	3.703 ± 0.023
unsloth 3.0	UD-Q4_K_XL	16746	16411	0.11012 ± 0.00258	3.950 ± 0.082	0.0225	2.3869	18.8515	95.89 ± 0.04	3.770 ± 0.023
bartowski	Q4_K_M	16949	16721	0.11468 ± 0.00262	4.188 ± 0.078	0.0320	2.5264	18.8423	95.23 ± 0.05	3.822 ± 0.024
unsloth 3.0 preview	UD-Q4_K_XL	17093	16821	0.10611 ± 0.00252	3.925 ± 0.080	0.0230	2.2947	18.6469	95.88 ± 0.04	3.772 ± 0.023
NikiKrutan	NIKI-Q5_K_S	17300	17085	0.09718 ± 0.00239	3.641 ± 0.079	0.0191	1.8434	18.0073	96.10 ± 0.04	3.780 ± 0.023
AtomicChat	AD-Q5_K-Q4_K	17688	17411	0.09972 ± 0.00240	3.907 ± 0.081	0.0216	1.9442	18.0818	96.01 ± 0.04	3.730 ± 0.023
unsloth 3.0	UD-Q5_K_S	17801	17466	0.10612 ± 0.00251	3.671 ± 0.082	0.0170	2.3331	18.3369	96.18 ± 0.04	3.796 ± 0.024
bartowski	Q4_K_L	17849	17621	0.11493 ± 0.00260	4.304 ± 0.079	0.0311	2.7333	18.9470	95.34 ± 0.05	3.822 ± 0.024
byteshape	Q5_K_M-5.60bpw	18249	17965	0.09784 ± 0.00241	3.863 ± 0.085	0.0164	1.9265	17.9692	96.35 ± 0.04	3.715 ± 0.023
unsloth 3.0 preview	Q5_K_S	18377	18080	0.10613 ± 0.00253	3.784 ± 0.082	0.0192	2.2872	18.5994	96.03 ± 0.04	3.739 ± 0.023
NikiKrutan	NIKI-Q5_K_M	18600	18385	0.08612 ± 0.00224	3.372 ± 0.082	0.0135	1.4814	17.2961	96.57 ± 0.04	3.792 ± 0.024
bartowski	Q5_K_S	18769	18541	0.09956 ± 0.00244	3.644 ± 0.081	0.0177	1.9769	18.5421	96.09 ± 0.04	3.798 ± 0.024
unsloth 3.0	UD-Q5_K_M	18856	18521	0.09175 ± 0.00233	3.449 ± 0.082	0.0132	1.6733	17.4314	96.60 ± 0.04	3.789 ± 0.024
unsloth 3.0 preview	Q5_K_M	18915	18606	0.10288 ± 0.00251	3.644 ± 0.082	0.0172	2.0636	18.6094	96.23 ± 0.04	3.783 ± 0.024
unsloth 3.0 preview	UD-Q5_K_XL	19282	18958	0.08932 ± 0.00230	3.423 ± 0.082	0.0135	1.5523	17.6310	96.52 ± 0.04	3.806 ± 0.024
AtomicChat	AD-Q5_K	19295	18990	0.08701 ± 0.00225	3.311 ± 0.081	0.0126	1.4779	17.3852	96.63 ± 0.04	3.760 ± 0.023
bartowski	Q5_K_M	19791	19563	0.09182 ± 0.00233	3.745 ± 0.086	0.0142	1.6026	17.6914	96.47 ± 0.04	3.757 ± 0.023
unsloth 3.0	UD-Q5_K_XL	19910	19575	0.09123 ± 0.00232	3.344 ± 0.084	0.0112	1.6803	17.6995	96.88 ± 0.04	3.790 ± 0.024
NikiKrutan	NIKI-Q6_K	20000	19785	0.07126 ± 0.00200	3.306 ± 0.087	0.0085	1.0398	15.9125	97.17 ± 0.04	3.772 ± 0.024
bartowski	Q5_K_L	20540	20312	0.08929 ± 0.00231	3.458 ± 0.084	0.0137	1.4471	17.7038	96.67 ± 0.04	3.779 ± 0.024
unsloth 3.0	UD-Q6_K	20965	20631	0.08346 ± 0.00219	3.176 ± 0.087	0.0086	1.3687	17.0122	97.13 ± 0.04	3.754 ± 0.023
lmstudio-community	Q6_K	21392	21060	0.07748 ± 0.00211	3.277 ± 0.083	0.0105	1.1575	16.4780	96.89 ± 0.04	3.734 ± 0.023
NikiKrutan	NIKI-Q6_K	21400	21185	0.05081 ± 0.00163	2.581 ± 0.083	0.0054	0.4680	13.4905	97.68 ± 0.03	3.770 ± 0.023
unsloth 3.0 preview	Q6_K	21824	21480	0.05497 ± 0.00169	2.794 ± 0.082	0.0069	0.5595	13.8988	97.36 ± 0.04	3.790 ± 0.024
AtomicChat	AD-Q6_K-Q5_K	22012	21689	0.08001 ± 0.00217	3.097 ± 0.083	0.0093	1.2233	16.9064	97.14 ± 0.04	3.772 ± 0.023
unsloth 3.0	UD-Q6_K_M	22019	21684	0.07816 ± 0.00211	3.066 ± 0.087	0.0069	1.1757	16.4711	97.36 ± 0.04	3.732 ± 0.023
bartowski	Q6_K	22376	22148	0.04659 ± 0.00156	2.447 ± 0.079	0.0054	0.3869	12.4783	97.58 ± 0.03	3.756 ± 0.023
bartowski	Q6_K_L	22963	22736	0.04494 ± 0.00151	2.301 ± 0.079	0.0046	0.3704	11.9331	97.87 ± 0.03	3.768 ± 0.023
NikiKrutan	NIKI-Q6_K_L	23000	22785	0.03539 ± 0.00132	2.125 ± 0.079	0.0037	0.2434	10.4939	98.06 ± 0.03	3.782 ± 0.024
unsloth 3.0	UD-Q6_K_L	23073	22738	0.06866 ± 0.00198	2.797 ± 0.085	0.0055	0.8841	15.8937	97.66 ± 0.03	3.753 ± 0.023
AtomicChat	AD-Q6_K	23847	23493	0.04172 ± 0.00146	2.283 ± 0.082	0.0040	0.3349	11.6762	98.00 ± 0.03	3.785 ± 0.024
unsloth 3.0	UD-Q6_K_XL	24127	23792	0.06625 ± 0.00196	2.756 ± 0.087	0.0046	0.7587	15.7494	97.78 ± 0.03	3.758 ± 0.023
unsloth 3.0 preview	UD-Q6_K_XL	24723	24334	0.03966 ± 0.00145	2.136 ± 0.081	0.0031	0.2803	11.7220	98.14 ± 0.03	3.768 ± 0.023

independent-v2 corpus

Warning! Huge table inside!

Lower is better except same top p

Qwen3.8-27B — independent-v2 corpus

Source	Label	Size, MiB*	Size w/o MTP, MiB	Mean KLD	RMS Δp	KLD p95	KLD p99	KLD p99.9	Same top p, %	PPL
unsloth 3.0	UD-IQ1_S	6240 *	5905	0.60813 ± 0.00186	23.522 ± 0.069	2.1155	4.1158	7.1822	66.99 ± 0.11	12.776 ± 0.078
unsloth 3.0	UD-IQ1_M	6752 *	6417	0.42429 ± 0.00145	19.376 ± 0.065	1.4844	3.2142	6.3670	72.03 ± 0.10	10.723 ± 0.064
unsloth 3.0	UD-IQ2_XXS	7264 *	6929	0.31338 ± 0.00120	16.361 ± 0.062	1.1123	2.5462	5.5458	75.58 ± 0.10	9.676 ± 0.058
AtomicChat	AD-IQ1_M	8104	7826	0.28422 ± 0.00107	15.475 ± 0.060	0.9671	2.2719	5.1406	76.70 ± 0.10	9.390 ± 0.055
unsloth 3.0	UD-IQ2_S	8319 *	7984	0.17177 ± 0.00080	11.770 ± 0.057	0.6043	1.5694	4.2142	81.83 ± 0.09	8.561 ± 0.051
NikiKrutan	NIKI-IQ2_XXS	8500	8316	0.18872 ± 0.00083	12.664 ± 0.058	0.6395	1.6962	4.4760	80.75 ± 0.09	8.639 ± 0.050
AtomicChat	AD-IQ2_XXS	8561	8283	0.21409 ± 0.00088	13.325 ± 0.057	0.7081	1.8009	4.7080	79.50 ± 0.09	8.848 ± 0.052
unsloth 3.0 preview	UD-IQ2_XXS	8593	8382	0.17763 ± 0.00079	12.199 ± 0.058	0.5790	1.5806	4.3460	80.77 ± 0.09	8.656 ± 0.051
bartowski	IQ2_XXS	8958	8730	0.26214 ± 0.00105	15.046 ± 0.062	0.8957	2.2519	5.2075	77.56 ± 0.09	9.283 ± 0.055
NikiKrutan	NIKI-IQ2_S	9100	8925	0.13775 ± 0.00068	10.720 ± 0.056	0.4627	1.2822	3.8480	83.43 ± 0.08	8.307 ± 0.049
unsloth 3.0	UD-Q2_K_XL	9374	9039	0.10854 ± 0.00059	9.232 ± 0.053	0.3701	1.0421	3.4345	85.33 ± 0.08	8.142 ± 0.048
AtomicChat	AD-IQ2_XS	9432	9153	0.14253 ± 0.00069	10.673 ± 0.053	0.4686	1.2057	3.6325	83.14 ± 0.08	8.336 ± 0.049
bartowski	IQ2_XS	9524	9296	0.19347 ± 0.00085	12.798 ± 0.058	0.6536	1.7233	4.5241	80.57 ± 0.09	8.736 ± 0.051
AtomicChat	AD-IQ2_S-IQ2_XS	9748	9469	0.11888 ± 0.00059	9.742 ± 0.052	0.3880	1.0273	3.2534	84.46 ± 0.08	8.171 ± 0.048
NikiKrutan	NIKI-IQ3_XXS	9800	9622	0.10149 ± 0.00053	9.141 ± 0.052	0.3371	0.9496	3.0609	85.73 ± 0.08	8.054 ± 0.047
bartowski	IQ2_S	9818	9590	0.17692 ± 0.00081	12.181 ± 0.058	0.5928	1.5965	4.4122	81.30 ± 0.09	8.616 ± 0.051
unsloth 3.0 preview	UD-IQ2_M	9842	9636	0.10846 ± 0.00054	9.425 ± 0.052	0.3404	0.9451	3.1076	84.34 ± 0.08	8.168 ± 0.048
unsloth 3.0 preview	UD-Q2_K_XL	10182	9976	0.09559 ± 0.00049	8.815 ± 0.050	0.2962	0.8225	2.8128	85.05 ± 0.08	8.074 ± 0.048
bartowski	IQ2_M	10370	10142	0.12842 ± 0.00061	10.298 ± 0.054	0.4225	1.1543	3.5092	83.75 ± 0.08	8.262 ± 0.048
unsloth 3.0	UD-IQ3_XXS	10428	10094	0.07049 ± 0.00044	7.453 ± 0.050	0.2356	0.6935	2.3984	87.90 ± 0.07	7.894 ± 0.046
NikiKrutan	NIKI-IQ3_XXS	10500	10322	0.07886 ± 0.00043	8.066 ± 0.047	0.2452	0.6858	2.4708	87.33 ± 0.07	7.895 ± 0.045
AtomicChat	AD-IQ2_S	10626	10347	0.08589 ± 0.00046	8.298 ± 0.047	0.2786	0.7362	2.4607	86.99 ± 0.08	7.958 ± 0.046
mudler	APEX-I-Nano	10720	10289	0.09942 ± 0.00053	8.882 ± 0.053	0.3395	0.9385	3.0039	86.06 ± 0.08	8.091 ± 0.048
byteshape	IQ3_S-3.44bpw	11226	11058	0.06952 ± 0.00044	7.384 ± 0.051	0.2318	0.7017	2.6157	88.54 ± 0.07	7.894 ± 0.046
NikiKrutan	NIKI-IQ3_S	11300	11116	0.05702 ± 0.00035	6.788 ± 0.046	0.1825	0.5295	1.9456	89.37 ± 0.07	7.783 ± 0.045
unsloth 3.0 preview	UD-IQ3_XXS	11362	11167	0.05629 ± 0.00037	6.829 ± 0.049	0.1801	0.5481	2.1643	89.45 ± 0.07	7.837 ± 0.046
unsloth 3.0	UD-IQ3_S	11483	11148	0.04598 ± 0.00031	5.902 ± 0.044	0.1538	0.4554	1.6954	90.34 ± 0.07	7.735 ± 0.045
AtomicChat	AD-IQ3_XXS	11516	11238	0.05924 ± 0.00034	6.850 ± 0.043	0.1870	0.5114	1.8070	89.11 ± 0.07	7.795 ± 0.045
byteshape	IQ4_XS-3.67bpw	11975	11806	0.05568 ± 0.00038	6.634 ± 0.050	0.1842	0.5734	2.1963	89.73 ± 0.07	7.804 ± 0.046
bartowski	IQ3_XXS	12042	11814	0.06745 ± 0.00040	7.340 ± 0.048	0.2191	0.6406	2.3279	88.23 ± 0.07	7.862 ± 0.046
NikiKrutan	NIKI-IQ3_S	12100	11890	0.04349 ± 0.00031	5.962 ± 0.047	0.1365	0.4072	1.7680	90.25 ± 0.07	7.746 ± 0.045
AtomicChat	AD-IQ3_S-IQ3_XXS	12383	12104	0.03803 ± 0.00026	5.393 ± 0.040	0.1243	0.3376	1.3080	90.88 ± 0.07	7.687 ± 0.045
unsloth 3.0	UD-Q3_K_XL	12537	12203	0.03045 ± 0.00023	4.843 ± 0.041	0.1006	0.2859	1.1419	92.00 ± 0.06	7.657 ± 0.045
bartowski	IQ3_XS	12713	12485	0.05041 ± 0.00031	6.326 ± 0.042	0.1600	0.4567	1.6053	89.92 ± 0.07	7.747 ± 0.045
unsloth 3.0 preview	UD-Q3_K_XL	12818	12620	0.03249 ± 0.00024	5.156 ± 0.041	0.1015	0.2971	1.2002	91.63 ± 0.06	7.687 ± 0.045
NikiKrutan	NIKI-IQ4_XS	13000	12790	0.03034 ± 0.00025	4.923 ± 0.042	0.0942	0.2761	1.3123	91.81 ± 0.06	7.663 ± 0.045
byteshape	IQ4_XS-4.00bpw	13030	12835	0.03808 ± 0.00028	5.450 ± 0.045	0.1261	0.4026	1.5728	91.46 ± 0.06	7.707 ± 0.045
AtomicChat	AD-IQ3_S	13197	12971	0.02895 ± 0.00021	4.662 ± 0.036	0.0957	0.2545	0.9263	92.02 ± 0.06	7.666 ± 0.045
bartowski	IQ3_M	13259	13032	0.04607 ± 0.00028	6.120 ± 0.041	0.1435	0.4139	1.5318	90.53 ± 0.07	7.750 ± 0.045
mudler	APEX-I-Mini	13297	12867	0.04092 ± 0.00030	5.675 ± 0.045	0.1354	0.4037	1.5444	91.03 ± 0.06	7.725 ± 0.046
unsloth 3.0	UD-IQ4_XS	13593	13258	0.02075 ± 0.00018	4.029 ± 0.040	0.0662	0.1947	0.9067	93.34 ± 0.06	7.605 ± 0.044
AtomicChat	AD-IQ4_XS-IQ3_S	13769	13534	0.02314 ± 0.00017	4.180 ± 0.036	0.0744	0.2036	0.8089	92.82 ± 0.06	7.635 ± 0.045
NikiKrutan	NIKI-IQ4_XS	14000	13790	0.01777 ± 0.00016	3.754 ± 0.038	0.0552	0.1621	0.6999	93.75 ± 0.06	7.600 ± 0.045
byteshape	IQ4_XS-4.40bpw	14344	14115	0.01946 ± 0.00017	3.825 ± 0.034	0.0639	0.1949	0.7830	93.69 ± 0.06	7.588 ± 0.044
unsloth 3.0	UD-Q4_K_S	14647	14312	0.01431 ± 0.00015	3.376 ± 0.038	0.0453	0.1370	0.6510	94.58 ± 0.05	7.574 ± 0.044
bartowski	IQ4_XS	14847	14619	0.01530 ± 0.00015	3.445 ± 0.036	0.0480	0.1374	0.5749	94.27 ± 0.05	7.588 ± 0.045
unsloth 3.0 preview	IQ4_XS	14978	14725	0.01467 ± 0.00015	3.339 ± 0.034	0.0466	0.1319	0.5580	94.33 ± 0.05	7.584 ± 0.044
NikiKrutan	NIKI-IQ4_XS	15000	14785	0.01448 ± 0.00015	3.383 ± 0.035	0.0453	0.1371	0.6066	94.38 ± 0.05	7.578 ± 0.044
byteshape	Q5_K_S-4.72bpw	15394	15144	0.01562 ± 0.00015	3.447 ± 0.036	0.0505	0.1536	0.6493	94.31 ± 0.05	7.571 ± 0.044
unsloth 3.0	UD-Q4_K_M	15702	15367	0.01037 ± 0.00013	2.832 ± 0.032	0.0331	0.0952	0.4359	95.33 ± 0.05	7.558 ± 0.044
AtomicChat	AD-IQ4_XS	15748	15490	0.01102 ± 0.00013	2.886 ± 0.032	0.0349	0.0989	0.4484	95.10 ± 0.05	7.556 ± 0.044
bartowski	Q4_K_S	15939	15711	0.01323 ± 0.00013	3.163 ± 0.031	0.0424	0.1194	0.4670	94.50 ± 0.05	7.571 ± 0.044
lmstudio-community	Q4_K_M	16032	15781	0.01954 ± 0.00017	3.796 ± 0.034	0.0631	0.1724	0.6741	93.41 ± 0.06	7.601 ± 0.045
NikiKrutan	NIKI-IQ4_XS	16100	15885	0.01211 ± 0.00014	3.070 ± 0.032	0.0373	0.1066	0.4506	94.69 ± 0.05	7.564 ± 0.044
unsloth 3.0 preview	Q4_K_M	16314	16038	0.01211 ± 0.00013	3.017 ± 0.030	0.0384	0.1088	0.4570	94.75 ± 0.05	7.570 ± 0.044
AtomicChat	AD-Q4_K	16328	16062	0.01002 ± 0.00013	2.769 ± 0.031	0.0319	0.0876	0.3724	95.26 ± 0.05	7.555 ± 0.044
unsloth 3.0	UD-Q4_K_XL	16746	16411	0.00768 ± 0.00012	2.410 ± 0.029	0.0242	0.0705	0.3030	95.88 ± 0.04	7.546 ± 0.044
bartowski	Q4_K_M	16949	16721	0.01058 ± 0.00012	2.864 ± 0.032	0.0332	0.0953	0.4210	95.09 ± 0.05	7.563 ± 0.044
unsloth 3.0 preview	UD-Q4_K_XL	17093	16821	0.00822 ± 0.00012	2.507 ± 0.030	0.0255	0.0754	0.3284	95.74 ± 0.05	7.548 ± 0.044
NikiKrutan	NIKI-Q5_K_S	17300	17085	0.00745 ± 0.00012	2.396 ± 0.029	0.0232	0.0687	0.2908	95.96 ± 0.04	7.539 ± 0.044
AtomicChat	AD-Q5_K-Q4_K	17688	17411	0.00687 ± 0.00012	2.297 ± 0.031	0.0216	0.0593	0.2554	96.01 ± 0.04	7.540 ± 0.044
unsloth 3.0	UD-Q5_K_S	17801	17466	0.00569 ± 0.00010	2.102 ± 0.028	0.0179	0.0520	0.2294	96.40 ± 0.04	7.539 ± 0.044
bartowski	Q4_K_L	17849	17621	0.00997 ± 0.00013	2.753 ± 0.030	0.0319	0.0903	0.3688	95.34 ± 0.05	7.560 ± 0.044
byteshape	Q5_K_M-5.60bpw	18249	17965	0.00675 ± 0.00011	2.248 ± 0.030	0.0211	0.0619	0.2567	96.12 ± 0.04	7.540 ± 0.044
unsloth 3.0 preview	Q5_K_S	18377	18080	0.00575 ± 0.00009	2.109 ± 0.026	0.0178	0.0494	0.2115	96.25 ± 0.04	7.534 ± 0.044
NikiKrutan	NIKI-Q5_K_M	18600	18385	0.00468 ± 0.00010	1.879 ± 0.025	0.0144	0.0422	0.1810	96.73 ± 0.04	7.527 ± 0.044
bartowski	Q5_K_S	18769	18541	0.00527 ± 0.00010	1.982 ± 0.026	0.0163	0.0438	0.1773	96.41 ± 0.04	7.533 ± 0.044
unsloth 3.0	UD-Q5_K_M	18856	18521	0.00427 ± 0.00010	1.822 ± 0.030	0.0129	0.0366	0.1589	96.78 ± 0.04	7.527 ± 0.044
unsloth 3.0 preview	Q5_K_M	18915	18606	0.00492 ± 0.00009	1.921 ± 0.027	0.0151	0.0402	0.1639	96.48 ± 0.04	7.530 ± 0.044
unsloth 3.0 preview	UD-Q5_K_XL	19282	18958	0.00415 ± 0.00009	1.766 ± 0.026	0.0126	0.0353	0.1452	96.80 ± 0.04	7.530 ± 0.044
AtomicChat	AD-Q5_K	19295	18990	0.00391 ± 0.00006	1.747 ± 0.026	0.0118	0.0331	0.1490	96.88 ± 0.04	7.526 ± 0.044
bartowski	Q5_K_M	19791	19563	0.00439 ± 0.00011	1.837 ± 0.031	0.0133	0.0356	0.1430	96.68 ± 0.04	7.529 ± 0.044
unsloth 3.0	UD-Q5_K_XL	19910	19575	0.00322 ± 0.00010	1.550 ± 0.025	0.0101	0.0278	0.1239	97.32 ± 0.04	7.525 ± 0.044
NikiKrutan	NIKI-Q6_K	20000	19785	0.00280 ± 0.00005	1.462 ± 0.021	0.0088	0.0250	0.1108	97.51 ± 0.04	7.519 ± 0.044
bartowski	Q5_K_L	20540	20312	0.00402 ± 0.00011	1.738 ± 0.029	0.0125	0.0337	0.1400	96.97 ± 0.04	7.527 ± 0.044
unsloth 3.0	UD-Q6_K	20965	20631	0.00244 ± 0.00010	1.353 ± 0.029	0.0074	0.0207	0.0885	97.65 ± 0.03	7.521 ± 0.044
lmstudio-community	Q6_K	21392	21060	0.00267 ± 0.00003	1.419 ± 0.020	0.0081	0.0210	0.0851	97.29 ± 0.04	7.521 ± 0.044
NikiKrutan	NIKI-Q6_K	21400	21185	0.00192 ± 0.00008	1.192 ± 0.019	0.0056	0.0162	0.0714	97.93 ± 0.03	7.519 ± 0.044
unsloth 3.0 preview	Q6_K	21824	21480	0.00209 ± 0.00010	1.224 ± 0.019	0.0059	0.0153	0.0603	97.58 ± 0.04	7.522 ± 0.044
AtomicChat	AD-Q6_K-Q5_K	22012	21689	0.00248 ± 0.00004	1.404 ± 0.030	0.0077	0.0217	0.0945	97.67 ± 0.03	7.521 ± 0.044
unsloth 3.0	UD-Q6_K_M	22019	21684	0.00198 ± 0.00010	1.210 ± 0.030	0.0059	0.0168	0.0710	97.99 ± 0.03	7.519 ± 0.044
bartowski	Q6_K	22376	22148	0.00176 ± 0.00009	1.136 ± 0.016	0.0050	0.0129	0.0506	97.69 ± 0.03	7.517 ± 0.044
bartowski	Q6_K_L	22963	22736	0.00142 ± 0.00007	1.001 ± 0.015	0.0042	0.0117	0.0454	98.19 ± 0.03	7.516 ± 0.044
NikiKrutan	NIKI-Q6_K_L	23000	22785	0.00124 ± 0.00002	0.986 ± 0.024	0.0038	0.0106	0.0428	98.23 ± 0.03	7.515 ± 0.044
unsloth 3.0	UD-Q6_K_L	23073	22738	0.00163 ± 0.00010	1.113 ± 0.035	0.0046	0.0133	0.0574	98.18 ± 0.03	7.517 ± 0.044
AtomicChat	AD-Q6_K	23847	23493	0.00105 ± 0.00002	0.889 ± 0.021	0.0032	0.0087	0.0344	98.45 ± 0.03	7.517 ± 0.044
unsloth 3.0	UD-Q6_K_XL	24127	23792	0.00139 ± 0.00009	1.029 ± 0.034	0.0039	0.0111	0.0504	98.34 ± 0.03	7.517 ± 0.044
unsloth 3.0 preview	UD-Q6_K_XL	24723	24334	0.00102 ± 0.00007	0.851 ± 0.011	0.0029	0.0083	0.0359	98.41 ± 0.03	7.515 ± 0.044

MTP

I've tested MTP performance and discovered that there is no point to keep MTP head higher than IQ4_XS. Q3_K is near but slightly worse (used for lower quants). Exact allocation of IQ4_XS/Q3_K is chosen by allocator (not pinned). Lower than that there is considerable quality drop, so not used in my quants. Higher is just waste of model size and VRAM.

For numbers check my previous GRM-2.6-Plus-0628 quant's MTP section.

I've done fast tests on other models (same architecture). Same conclusions.

Also I've additionally checked fast if there is any degradation on IQ4_XS vs Q8_0 with longer context (tested up to 28K), none found. And temperature = 0.6, still no evidence of degradation. But I must say that doing this test properly and scientifically much time and effort is needed. Since not 0 temperature gives different results: many runs needed to stabilize. Reasonably long context (100K+ from my opinion) is very slow on my hardware. So there actually may be some degradation. But that should be proven and not taken by "it seems" or "everybody knows". If there is such comprehensive analysis already, please let me know.

Basically I use these parameters for MTP: --spec-draft-n-max 5 --spec-draft-p-min 0.8. Primary reason of poor results with MTP is ignoring --spec-draft-p-min. Don't put obvious crap in your drafts. But let drafter do more if it is sure enough. MTP performance speed-up is highly dependable on hardware, configuration and specific context. So to decide what is best for your case you should test yourself.

Also don't use other llama.cpp drafters like ngram-mod with MTP. MTP wastes time even if ngram-mod already done draft. That is how llama.cpp is programmed. Not obvious. I have created experimental fork to overcome this issue (and added much better ngram-mod-v2 + minor fixes): https://github.com/NikiKrutan/niki-llama.cpp. I use it myself on a daily basis, but it is more like crude draft than real working fork. It breaks some llama.cpp behavior. But it gives another ~1.5x speed-up on top of MTP for my use cases.

Chat template

Chat template included with my gguf's is v22.2 from froggeric. There may be newer version as it is often updated.

If you have any problems with it you may also try one from Moore2877.

Multimodality

I've quantized mmproj following DAXZEIT's receipt for Qwen3.6 that I used in Qwen3.6 era.

It reduces mmproj size saving practically same quality. Why reducing size? Well, it is probably not needed for most use cases. But with this I can put my 17300 quant with q8_0 K-cache and q4_0 V-cache + MTP with the same KV-cache quantization + mmproj + 262K full context in my 2x16Gb VRAM. With BF16 mmproj I can't do it despite very little difference in size. Every MiB matters when you are about quality.

Also any standard Qwen3.8-27B mmproj should work if you want full BF16 one.

Acknowledgments

Qwen - great base model. Really great!
llama.cpp - quantization and evaluation framework.
unsloth - quants for comparison.
AtomicChat - quants for comparison.
bartowski - quants for comparison.
lmstudio-community - quants for comparison.
mudler - quants for comparison.
byteshape - quants for comparison.
FineWeb - data for evaluation dataset.
caselaw_access_project - data for evaluation dataset.
open-web-math - data for evaluation dataset.
recursal - data for evaluation dataset.
DAXZEIT - mmproj receipt.
michaelw9999 - some ideas to start with.
Required attributions for exact science papers used for evaluation dataset are on dataset page.

Downloads last month: 5,118

GGUF

Model size

27B params

Architecture

qwen35

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

Model tree for NikiKrutan/Qwen3.8-27B-MTP-GGUF

Base model

Qwen/Qwen3.8-27B

Quantized

(1193)

this model