unsloth

/

Qwen3.8-Flash-Next-GGUF

like 800

Follow

Unsloth AI 34.4k

the architecture is completely different. in llama.cpp it's called qwen4exp (i assume experimental). look at the graph of the model architecture. sot the models you mention are not comparable, even if your reasoning about active parameters seems intuitive. however i get 23 t/s with no MTP (strix halo same as yours) with the latest llama.cpp release

What are your both configs @auf1r2 and @dilavni ?

i use json files to configure my engines, so sorry for the non-standard presentation. (also mtp parameters are not valid as the draft type is none)

Both numbers are probably right - the decode speed is context-depth-dependent, not a config difference. Measured on 3x RTX 3090 (UD-Q3_K_XL): 17 ms/tok (59 t/s) at 2K depth, falling to 70 ms/tok (14 t/s) once the context is deep (140K). The same cliff exists on Strix Halo (ggml-org/llama.cpp#27856).

So dilavni-s 23 t/s is the shallow-context speed and auf1r2-s ~5-6 t/s is the deep-context speed - same build, different depth. During deep-context decode the GPUs sit at 0% utilization while the host burns ~85% of one CPU core (per-token block-table rescans in set_input_qsa).

Full depth curve, host-CPU measurements and fix direction: discussion #40.

Ok, I have update on this. The problem was with --flash-attn on - removing this gave me same 22t/s on Vulkan as folks reported above.

My current config for Vulkan is (on Ubuntu 26.04 x64):

# GENERIC
--host 192.168.1.86
--port 8080
--tools get_info
--api-key XXX
--jinja
--gpu-layers all
--parallel 1
--no-kv-unified
--kv-offload
--load-mode none
--image-min-tokens 2048

# MEMORY
--cache-type-k q8_0
--cache-type-v q8_0
--ctx-size 262144
--batch-size 4096
--ubatch-size 2048

# MODEL
--temperature 0.8
--top-k 20
--repeat-penalty 1.05
--presence-penalty 0.05
--top-p 0.95
--min-p 0.05
--reasoning on
--chat-template-kwargs '{"reasoning_effort":"xhigh"}'

But dispite this I still see some issues in current llama.cpp version b10666 Q4_K_XL quant from this repo.

1. On ROCm (never on Vulkan) it has some troubles with reasoning, can't say exactly what causing this but sometimes reasoning starts typing infinite amount of back-slashes like

Reasoning:
/////////////////////////////////////////////////////////////////////////////////////////////////   <== until context fills up fully 

2. --flash-attn on for Vulkan drops tg twice - not sure bug or feature.
3. MTP not working - I assume this is expected for now, since PR still not merged
4. Even 22t/s still looks slow for A6B model, I would expect 35+ t/s on this hardware

Good thing 😀
I tried 160GB Q5_K_XL on my Strix Halo 128 Gb and it works sligthy slower, not as slow as you would expect.
I expected like 2-3x slower, but on clean contex it starts from 19t/s tg / 240t/s pp and slowly degrades as context growhs.

So this made me optimistic to download Q6_K_XL 170GB and try it.

might not be relevat: i build the gglm llama.cpp release tags my self and they use my system ROCm 7.2.3, which is slightly newer than the gglm rocm releases.

For some reason I think everybody is getting something around 10 to 20 t/s with this model. Even I, with a 3080 (10gb) am getting 16t/s with 3bit. Ill test 4bit later.

Epyc 7402p / 2933MHz RAM (8 channels) + 1x 3090 I get 30 t/s with the Q4.

Single RTX 3090 with 130k context using UD-IQ4-XS quant gave me:

• pp 145 t/s
• gen 12 t/s

I did not load any layer into the RAM, ssd + gpu only.

If I load the remaining into RAM, then it gives:

• pp 300 t/s
• gen 20 t/s

i also tried with vulkan back-end: the (shallow) ts is 21 (down 2 ts from rocm). not much, but above noise.

Have you tried it via unsloth? It's slightly faster.

There's still lots of improvements to be made which were gonna investigate.

Unsloth is in no way faster. I downloaded their latest version, all same settings, unsloth about 4-5 t/s. Official llama 13.40 t/s. Then I enabled mtp on unsloth, hoping it will be better than official, but it wasn't. Unsloth is just broken fork or they optimized it only for certain gpus.

That's not true. I'm running with unsloth llama.cpp build with the mtp support and hitting 36ts at 10K (well over 40 at jump) on vulkan backend on Strix Halo.