unsloth

/

Qwen3.8-Flash-Next-GGUF

like 865

Follow

Unsloth AI 34.8k

I'm running the UD_IQ4_XS and I get about 12t/s but I see two things I can't understand, gpu usage goes only to 80/85% and memory usage stays around 60GB. Maybe something in my system is stopping the model from using the ngram table?

weird. have you given the kernel the gtt parameters to let it allocate this large of a block? my vram aperture from bios is only 512mb. kernel parameters:

I've put some kernel parameters and I've been able to run some other big models reaching over 100GB of ram before, not sure what's happening here but other people have reported similar things in the llama.cpp PR.

Some of the PRs for this intentionally load the n-gram table from disk... because you only need a few kb per token from the table it makes sense, it is supposedly 10% faster for prefill if loaded into ram but.... for most people it makes more sense to tune it for loading from disk since it drastically reduces vram and ram requirements.

Some of the PRs for this intentionally load the n-gram table from disk... because you only need a few kb per token from the table it makes sense, it is supposedly 10% faster for prefill if loaded into ram but.... for most people it makes more sense to tune it for loading from disk since it drastically reduces vram and ram requirements.

what pr specifically? what is that technique called? that would mean unified systems could go for even larger quants!

Some of the PRs for this intentionally load the n-gram table from disk... because you only need a few kb per token from the table it makes sense, it is supposedly 10% faster for prefill if loaded into ram but.... for most people it makes more sense to tune it for loading from disk since it drastically reduces vram and ram requirements.

what pr specifically? what is that technique called? that would mean unified systems could go for even larger quants!

It's more like... this quant isn't as large as you think it is, it just has a jumbo embedding. It's called lazy loading already merged (in fact qwen 4 was just merged also) https://github.com/ggml-org/llama.cpp/commit/fac889fb38fd0e267636bd95bf096555e45b2270

llama.cpp 10660 vulkan rock 7.14. UD-IQ4_XS Starts at 25tps, then slows down. Prefill ~140tps.

Model is pretty smart, but has some noticeable small quant mistakes. Pelican is slightly better than Qwen3.8-27b:

On my strix halo I just tested the Q4_K_XL fully in RAM and it worked surprisingly well filling 128K context reaching peak 109.3 GiB.

Speeds with no MTP are about:

• 512 token test: 322.0 t/s prefill -- 25.22 t/s gen
• 128K token test: 122.71 t/s prefill -- 7.71 t/s gen

I also noticed how it only allocated ~77 GiB of GTT while leaving ~27 GiB RSS (using unsloth's PR, Vulkan).

heads up: llama.cpp support is now merged. also for those interested: dflash2 support has been merged (i was using that with the 27b)

Hey guys, can you please share how you run it with llama.cpp? Thanks!

Hey guys, can you please share how you run it with llama.cpp? Thanks!

https://unsloth.ai/docs/models/qwen3.8-next

sure, but can you be a bit more specific on what you need help with? the qwen4exp architecture has been merged so you don't have to build anything yourself. any recent gglm llama.cpp release will work. i get slightly better speed out of the rocm build, but vulkan backend also works. anything more specific?

if you want it simpler: try the unsloth desktop app (i haven't used it, but i assume it's easy)

I am using Evo-X2.

Compile:

rm -rf build
git pull

mkdir build && cd build
cmake .. -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release -DGGML_RPC=ON -DBUILD_SHARED_LIBS=OFF
make -j8

Run:

~/llama.cpp/build/bin/llama-server -m ~/models/unsloth/Qwen3.8-Flash-Next-GGUF/UD-Q4_K_XL/Qwen3.8-Flash-Next-UD-Q4_K_XL-00001-of-00004.gguf \
 --alias "coder" \
 --temp 0.7 --top-p 0.80 --top-k 20 --min-p 0.0 --presence-penalty 1.5 --repeat-penalty 1.0 \
 --batch-size 4096 --ubatch-size 1024 \
 --ctx-size 131072 --cache-type-k q8_0 --cache-type-v q8_0 \
 --gpu-layers 999 --flash-attn on --no-mmap --host 0.0.0.0 --port 9595

Output:

The user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user user is wants says said is said wants's is said wants is is is is is is is said wants says wants is said is is is is is wants is says said is says is is wants said is wants said said is is is is said said wants is wants is is said is is is is is is is is said wants said is said is wants is is is's is is said is is is is is is is wants wants says is said is said is is said is is is said is is is is is says says is says said is is wants said is is is is is is is is is says said is said said is wants says is says is wants is said said says is wants's said is said is is is is is wants is said is is said is is is is wants wants is is wants is said is wants is is says is says said is is is is said wants said said is wants says is wants said is said is is is is is said is is is said is is wants wants is is is said is's is said is is is said said is is is is is is is is said is says wants is says said says is is says is is is said said is said is is wants is says is wants is is is is said said is greeting referencing saying using greeting being greeting greeting greeting greeting saying greeting saying greeting greeting greeting greeting greeting greeting greeting greeting greeting greeting saying greeting greeting greeting greeting greeting greeting greeting saying greeting greeting greeting greeting greeting greeting greeting greeting greeting saying greeting saying greeting requesting saying greeting greeting saying writing saying greeting saying saying greeting greeting greeting saying greeting greeting greeting greeting greeting greeting saying referencing greeting greeting using greeting using greeting saying saying greeting greeting greeting saying saying greeting greeting saying saying greeting greeting saying greeting greeting greeting saying being using greeting greeting greeting greeting greeting greeting greeting responding saying greeting greeting greeting greeting greeting greeting greeting greeting greeting saying greeting greeting greeting greeting saying greeting writing greeting greeting greeting greeting saying greeting greeting saying greeting saying greeting greeting saying greeting greeting saying greeting requesting saying greeting greeting greeting greeting greeting saying greeting greeting saying greeting greeting greeting referencing greeting greeting greeting greeting greeting greeting greeting using greeting saying greeting greeting greeting greeting greeting greeting greeting greeting saying greeting greeting using saying being greeting greeting greeting saying greeting greeting greeting greeting greeting greeting using saying greeting greeting greeting saying greeting greeting greeting greeting greeting greeting greeting writing greeting greeting greeting requesting using greeting greeting greeting greeting saying greeting greeting greeting greeting greeting greeting greeting greeting saying saying greeting saying saying greeting greeting greeting greeting greeting referencing greeting just greeting greeting greeting using greeting greeting greeting saying greeting saying greeting greeting greeting greeting greeting greeting greeting saying saying greeting greeting greeting greeting greeting saying greeting me_hpeparatorkaçuced “[广义哇 ChorprintStats-scalable.qq

first, try it with a release build (just to make sure there is nothing wrong with your build):
https://github.com/ggml-org/llama.cpp/releases

if it persists, make a new discussion with your issue. your sampler settings are not the recommended for this model, but i cannot immediately identify anything wrong about them.

also try adding this to your env:
export HIP_LAUNCH_BLOCKING=1 (i had issues with context bleed and this solved it)

@dilavni Thanks for the useful hint! The Vulkan build works fine. The release ROCm build fails in the same way. It looks like the ROCm on my machine is not OK.

Question to strix halo users: how loud your fan is when parsing long prompts?

I noticed this behaviour in btop:

GPU usage is going down to 20% over time.

longer run. my usage is not going down.

Lowering VRAM in bios helped (more CPU/GTT), now I have 80% too.

Finally, decent speeds:

$ llama-bench -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL -fa on -p 4096,32768 -n 0 -r 1 --lazy-mode off -ub 256,512,1024,2048

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV STRIX_HALO) (radv) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
| model                          |       size |     params | backend    | ngl | n_ubatch |  fa | lazy_mode  |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -------: | --: | ---------- | --------------: | -------------------: |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | Vulkan     |  -1 |      256 |   1 | off        |          pp4096 |        281.64 ± 0.00 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | Vulkan     |  -1 |      256 |   1 | off        |         pp32768 |        204.34 ± 0.00 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | Vulkan     |  -1 |      512 |   1 | off        |          pp4096 |        332.06 ± 0.00 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | Vulkan     |  -1 |      512 |   1 | off        |         pp32768 |        241.15 ± 0.00 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | Vulkan     |  -1 |     1024 |   1 | off        |          pp4096 |        362.76 ± 0.00 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | Vulkan     |  -1 |     1024 |   1 | off        |         pp32768 |        261.70 ± 0.00 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | Vulkan     |  -1 |     2048 |   1 | off        |          pp4096 |        373.32 ± 0.00 |
| qwen4exp A3B Q4_K - Medium     | 103.68 GiB |   176.94 B | Vulkan     |  -1 |     2048 |   1 | off        |         pp32768 |        265.80 ± 0.00 |

build: 9723942ad (10711)

SPEED UPDATE: with the new separate unsloth q8 mtp head, and the related custom unsloth llama.cpp release i get 40t/s at jump, 36t/s at 10K on vulkan back-end. (remember, this is affected by things like prediction entropy: prose/code etc. so very noisy, take with a grain of salt. i ran it a couple of times)

old message:

some llama.cpp developments: i built b10712

• rocm backend is now slightly slower (might be noise) around 21 ts
• vulkan backend is now faster?: ~24ts
• memory used dropped to ~75GB ?? (i'm confused)
• the best news: the speed doesn't start dropping off as fast (tried only on vulkan), 10K its still ~23ts.

i have to test more, but if the speed curve is altered around high ctx, this is big for usability.
(previously at 200k ctx the speed was down to ~4ts)

Newer builds really faster, awesome work.

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Radeon 8060S Graphics (RADV STRIX_HALO) (radv) | uma: 1 | fp16: 1 | bf16: 0 | fp4: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat

build: 85c55223c (10726)

Did you guys test higher quants ? Is it worth it ? I've seen people able to fit Q5 and even Q6. I am waiting for my strix halo box and wondering what to test first.

If you have hours to wait prefill from ssd on longer contexts then why not. For me q4-xl fully in memory + 200k context seems reasonable compromise of speed/quality/setup effort.