Qwen3.8-27B UD-Q2_K_XL: unexpectedly slow ROCm prefill; ssm_alpha/ssm_beta are IQ1_M
I am seeing unexpectedly slow prompt processing with
Qwen3.8-27B-UD-Q2_K_XL.gguf on ROCm.
While inspecting the GGUF tensor table, I found that all 48
ssm_alpha.weight and 48 ssm_beta.weight tensors are quantized to IQ1_M.
The standard Q3 quants I inspected keep these tensors in F32.
Measurements
Hardware and software:
- Radeon 780M (gfx1103)
- ROCm 7.2
- llama.cpp b10333 (08659901c)
- Model fully offloaded with -ngl 99
Preliminary single-run measurements:
llama-bench -p 512 -n 32 -r 1 -fa 1 -ctk q8_0 -ctv q8_0
Configuration pp512 tg32
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━ ━━━━━━━
Qwen3.8-27B UD-Q2_K_XL, -ngl 99 13.60 4.54
───────────────────────────────── ──────── ───────
Qwen3.8-27B UD-Q2_K_XL, -ngl 0 13.26 3.33
───────────────────────────────── ──────── ───────
Qwen3.5-9B Q4_K_M, -ngl 99 318.22 13.71
The Qwen3.8 prompt-processing result is nearly unchanged between full GPU
offload and CPU execution. The Qwen3.5-9B control confirms that the same build
and GPU can achieve normal prefill performance on another qwen35 hybrid
model.
The 9B control is not a like-for-like comparison, however, so these results do
not by themselves isolate the cause.
Tensor inspection
I inspected the remote GGUF headers using HTTP range requests:
Quantization ssm_alpha / ssm_beta
━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━━━
UD-Q2_K_XL IQ1_M
────────────── ──────────────────────
UD-Q3_K_XL IQ4_XS
────────────── ──────────────────────
Q3_K_M F32
────────────── ──────────────────────
Q3_K_S F32
────────────── ──────────────────────
IQ4_XS F32
In llama.cpp b10333, I could not find a dedicated IQ1_M MMQ case in
ggml/src/ggml-cuda/mmq.cu or mmq.cuh, while IQ1_S and the dominant tensor
types used by this file do have MMQ cases.
I understand that the absence of an MMQ kernel does not necessarily mean CPU
fallback: llama.cpp also has a GPU dequantization path for IQ1_M. My current
hypothesis is therefore only that these IQ1_M projections may trigger a much
slower GPU path or an unfavorable scheduling boundary in the 48 DeltaNet
layers. I have not confirmed that hypothesis with a profiler or scheduler
trace.
Separately, quantizing these gating projections to approximately 1.75 bpw
saves only a small fraction of the model size—roughly 90 MB compared with
keeping them in F32—so this choice may also deserve review from a model-quality
perspective.
Questions
Is quantizing ssm_alpha and ssm_beta to IQ1_M intentional for
UD-Q2_K_XL?Has this variant been benchmarked with llama.cpp on ROCm?
Would it make sense to keep these projections in F32 for qwen35 hybrid
models, as the standard Q3 quants already do?
A same-model comparison against Qwen3.8-27B Q3_K_M or a requantization changing
only these tensors would be needed to establish causality. I have not performed
that comparison yet, but I can provide additional logs or test suggested
commands.
Did you try Vulkan?
That's really strange. I've also noticed Q2 embeddings and IQ1_M ssm.alpha/beta. It should definitely hit the performance (KL 99%, etc).
Qwen 3.6 27B UD Q2_K_XL was 12GB, while the current UD Q2_K_XL is about 10 GB. Something is wrong.
@anjeysapkovski Thanks — an independent confirmation is exactly what this report
was missing, since from a single machine I could not rule out that it was my
ROCm setup.
One correction on the size argument, because I think the two symptoms are worth
keeping apart: the ssm_alpha/ssm_beta tensors at IQ1_M account for
0.00 GiB of the file, so they explain none of the 12 GB → 10 GB drop. That
drop comes from the body of the model instead. Full type breakdown ofQwen3.8-27B-UD-Q2_K_XL.gguf (866 tensors):
| type | tensors | bytes | % of bytes |
|---|---|---|---|
| IQ3_XXS | 288 | 7.54 GiB | 75.9% |
| IQ3_S | 99 | 1.28 GiB | 12.9% |
| Q3_K | 1 | 0.51 GiB | 5.1% |
| Q2_K | 1 | 0.39 GiB | 3.9% |
| IQ4_XS | 21 | 0.20 GiB | 2.0% |
| F32 | 360 | 0.01 GiB | 0.1% |
| IQ1_M | 96 | 0.00 GiB | 0.0% |
So the file named UD-Q2_K_XL contains almost no Q2_K — it is mostly IQ3_XXS.
Both observations are consistent with the dynamic heuristic simply being more
aggressive on this release, but neither one is evidence for the other.
There is a second check that pins the IQ1_M issue down to the tensor: clean
quants of this model carry 456 F32 tensors, this file carries 360. The
difference is exactly 96 = 48 ssm_alpha + 48 ssm_beta — the DeltaNet
gating parameters of all 48 linear-attention layers.
Why that matters for speed specifically: IQ1_M has no MMQ kernel in
llama.cpp (zero occurrences in ggml/src/ggml-cuda/mmq.cu and mmq.cuh, whileIQ1_S does have one). Those tensors sit in the DeltaNet path, i.e. 48 of the
64 layers on every forward pass. That would explain the shape of what I measured
— batched prefill collapsing to CPU-level throughput while batch-1 generation,
which goes through dequantize_mul_mat_vec and covers every type, degrades much
less. To be clear, this is a coherent hypothesis, not something I isolated with
a profiler: confirming it properly would need a GGML_SCHED_DEBUG=2 build.
For anyone who just wants a working file in the meantime, I checked the tensor
metadata of the other quants without downloading them (the GGUF header is
enough, via a ranged HTTP request):
| quant | size | ssm_alpha/ssm_beta |
verdict |
|---|---|---|---|
| UD-Q2_K_XL | 10.7 GB | IQ1_M | affected |
| UD-Q3_K_XL | 13.4 GB | IQ4_XS | quantized, but IQ4_XS has an MMQ path — untested |
| Q3_K_M | 13.8 GB | F32 | clean |
| Q3_K_S | 12.6 GB | F32 | clean |
| IQ4_XS | 15.7 GB | F32 | clean |
I have been running Q3_K_M since, and it behaves normally on the same setup,
including the built-in MTP head (draft acceptance 0.76 at ~2.2k context and 0.96
at ~16k).
@itbj2 On Vulkan: no, I have not tested it, and I should be upfront that I
deleted the file after the diagnosis, so I would need to re-download ~10 GB to
answer. It is a fair question though — if the prefill collapse reproduces on
Vulkan, the cause is the missing kernel path for that type rather than anything
specific to ROCm. Happy to run it if it would help settle the issue.