Qwen

/

Qwen3.8-27B

like 14.2k

Follow

Qwen 103k

A while back I posted a comparison of Qwen3.8-27B against Nemotron 3.5 Lightning and Muse Glimmer on 16 hard problems. Several people asked what settings I was running, and it turned out I did not have a good answer. So I went back and benchmarked the settings themselves: 45 configurations of this model alone.

Sharing the findings here because a few of them changed how I run it, and one of them affects that earlier post.

Setup

• One RTX 5090, 32GB
• llama.cpp build b10448, CUDA, and Ollama 0.32.13 for the runtime comparison
• Q4_K_M GGUF (the Ollama blob, 27.3B), 32K context unless stated
• Median of 3 repeats, warmup discarded
• Server killed and GPU memory polled back to idle between every configuration, with the idle VRAM baseline re-measured each time

As before, this is about the 4-bit GGUF build, not the safetensors on this page.

1. The MTP draft head is real and it is free

Qwen3.8 ships a multi-token-prediction head inside the GGUF, an actual extra layer at index 64, present in a standard Q4_K_M download:

qwen35.nextn_predict_layers = 1
blk.64.nextn.eh_proj.weight   [10240, 5120]  Q4_K

One flag activates it. Draft depth is the only decision:

There is a peak because two curves fight: acceptance falls with depth (0.86 → 0.52) while tokens harvested per step rises (1.88 → 3.90). Product peaks at n=3.

Speculative decoding is lossless by construction, since a drafted token is kept only if the full model would have produced it, so nothing degrades at any depth. n=4 and n=5 are slower, not worse. Costs ~680MB at n=1, ~150MB per level after.

I re-ran the whole ladder in reverse order as a control, because this card is power-capped (599W of 600W, clocks dropping 3090 → 2700MHz) and a sweep run n=1 first would show a peak near the front regardless. n=3 still won. Acceptance rates came back identical to four decimals in both directions, which is the tell that the measurement is sound.

2. Quantizing the KV cache makes it faster, not slower

Decoding is memory-bandwidth bound, so a smaller cache means less data read per token. q4_0 is 1.4GB lighter and faster.

Worth noting: q4_0 at n=2 (136.7) beats f16 at n=3 (133.6), the best result from the entire draft-depth sweep. The cache type matters more than the draft depth. Also, the best depth changes with the cache. With q4_0, n=2 and n=3 tie, so n=3 just costs you memory.

q8_0 is about 63% of f16 in size, not 50%.

3. Reasoning is a bigger lever than any server flag

Time to the first word of actual answer improves ~10x. Turning reasoning off also improves draft acceptance (0.766 → 0.863), because reasoning prose is less predictable than a direct answer, so the effects multiply to 2.06x.

The chat template defines exactly three effort levels: low, medium, xhigh (default). medium cuts the wait by about a third for no measurable quality cost. llama.cpp's --reasoning-effort will also accept minimal, high and max; this model's template raises on them, so the server starts fine and then errors on every request.

4. Context: raising it is cheap, filling it is not

Decode holds around 126 tok/s from 4K to 64K, so a bigger window costs VRAM, not throughput. But filling it is superlinear: 128K → 256K multiplied time-to-first-token by 9.7x. 256K works on a 32GB card with perfect needle recall, but 12.5 minutes to first token makes it a batch setting. Interactive ceiling is 64K to 128K.

5. The one that affects my earlier post

Ollama measured ~2x a stock llama-server on the identical GGUF. That is not a runtime difference. Ollama runs llama-server as a child process. Reading the command line it passes:

--spec-type draft-mtp --spec-draft-n-max 4 --spec-draft-backend-sampling
-b 1024 -ub 1024 --no-jinja --chat-template chatml

Ollama enables the MTP draft head by default, at n=4. Adding its flags to stock llama.cpp one group at a time:

Matched on flags and template, llama.cpp is ~11% faster. There is no engine gap.

Why this matters for anyone benchmarking models on Ollama: it picks draft settings per model, silently. Checking the three models from my earlier post:

So that comparison had one model drafting 4 tokens, one drafting 2, and one with speculative decoding off entirely. The accuracy results there are unaffected, because speculation is lossless, but the speed and latency numbers were partly measuring Ollama's per-model defaults rather than the models. Worth checking ollama show --parameters <model> before trusting any cross-model throughput comparison.

One more Ollama note: because it discards the model's Jinja template in favour of chatml, reasoning_effort cannot be set through it at all.

What I run now

llama-server -m Qwen3.8-27B-Q4_K_M.gguf \
  -ngl 999 -fa on --jinja -np 1 \
  -c 32768 \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --spec-type draft-mtp --spec-draft-n-max 2

136.7 tok/s, 17.0GB, 100% quality, measured exactly as written. Add --reasoning-effort medium if you want answers sooner.

Not worth enabling: ngram-mod (0.97x on novel prompts, since its apparent gain is cache replay from repeated benchmark prompts), -fa off (costs 2.3GB and 20% of prefill), f16 KV cache (slower and larger than q4_0).

Caveats

One card, one Q4_K_M build, 3 repeats per config. Every quality task scored 100% including thinking-off and q4_0, which shows these settings do not break basic competence. It does not show reasoning is unnecessary, as the suite is not hard enough to test that.

Happy to share the harness or run specific configurations if anyone wants a case tested on this build.

Full write-up with all 12 charts, the method, and the four measurement traps that produced wrong numbers before I caught them:
https://kgptalkie.com/tutorials/generative-ai/qwen-3-8-27b-settings-that-matter

Earlier post in this thread (the 3-model comparison):
https://kgptalkie.com/tutorials/generative-ai/qwen-3-8-27b-vs-nemotron-3-5-vs-muse-glimmer