Why Qwen3.8-27B overthinks? Here the reason.
I checked the official GGUF BF16 weights directly from Unsloth.
What I found is not a “reasoning style” issue. It is a structural defect in the temporal processing layers.
| Tensor | QType | C2 | α | S_b | S_a |
|---|---|---|---|---|---|
| blk.52.ssm_conv1d.weight | F32 | ✓ | 0.59005 | 0.0016 | 0.0006 |
| blk.53.ssm_conv1d.weight | F32 | ✓ | 0.55484 | 0.0015 | 0.0005 |
| blk.56.ssm_conv1d.weight | F32 | ✓ | 0.54486 | 0.0015 | 0.0004 |
| blk.57.ssm_conv1d.weight | F32 | ✓ | 0.53574 | 0.0015 | 0.0004 |
| blk.58.ssm_conv1d.weight | F32 | ✓ | 0.60972 | 0.0012 | 0.0005 |
| blk.60.ssm_conv1d.weight | F32 | ✓ | 0.48136 | 0.0017 | 0.0004 |
| blk.61.ssm_conv1d.weight | F32 | ✓ | 0.65327 | 0.0010 | 0.0004 |
| blk.62.ssm_conv1d.weight | F32 | ✓ | 0.61856 | 0.0013 | 0.0005 |
Column meaning:
- C2 = scale misalignment detected
- α = optimal scale correction factor
- S_b = saturation before correction
- S_a = saturation after correction
In plain terms:
- these layers should prepare the signal for SSM recurrence
- instead, their scale is shifted far from the peer distribution
- α ≈ 0.48–0.65
That means the model does not receive a clean temporal signal. It receives a distorted one.
The result is exactly what users see:
- reasoning does not stop at the right time
- simple questions take 100+ seconds
- token consumption is ~5× higher than it should be
Here is a real example from the released model.
User asked:
I need to wash my car, the car wash is 100m away. Do I go by car or by foot?
The model spent 137 tokens in xhigh reasoning mode before answering:
By car - you need the car at the car wash to wash it.
That is not “too smart” or “thinking hard”. That is broken ssm_conv1d signal conditioning.
That is also why I will not make Genesis for 27B.
You cannot fix this by patching a few tensors or doing SVD to fix noise gate.
The SSM input pathway is damaged across too many layers.
A healthy model should answer:
By car - you need the car at the car wash.
Not spend 137 tokens explaining the obvious.
I really appreciate your attitude towards model tuning keep on,sir
What are the changes you made in the V8 (Hermes3.6-35B-A3B-Uncensored-Genesis-V8-APEX-Compact.gguf uploaded 1 hour ago) ? It would be cool to have a change list somewhere.
What are the changes you made in the V8 (Hermes3.6-35B-A3B-Uncensored-Genesis-V8-APEX-Compact.gguf uploaded 1 hour ago) ? It would be cool to have a change list somewhere.
Here info: https://huggingface.co/LuffyTheFox/Qwen3.6-35B-A3B-Uncensored-Genesis-Hermes-V7-GGUF/discussions/39
Independent verification of your conv1d diagnosis — full benchmark results
Thank you for this write-up — it prompted us to run an independent, controlled verification, and your diagnosis holds up. We are the hermes-nova team (org redashes), working with official Qwen/Qwen3.8-27B BF16 weights.
1. Diagnosis confirmed by independent measurement
We measured linear_attn.conv1d.weight std across the 48 GatedDeltaNet layers on the official BF16 weights, and flagged the same 8 layers using a std/median > 1.6 rule:
| Layer | our α | your α | match |
|---|---|---|---|
| 52 | 0.59005 | 0.59005 | ✓ |
| 53 | 0.55484 | 0.55484 | ✓ |
| 56 | 0.54486 | 0.54486 | ✓ |
| 57 | 0.53574 | 0.53574 | ✓ |
| 58 | 0.60972 | 0.60972 | ✓ |
| 60 | 0.48136 | 0.48136 | ✓ |
| 61 | 0.65327 | 0.65327 | ✓ |
| 62 | 0.61856 | 0.61856 | ✓ |
Layer set and α values are identical — two independent implementations, convergent diagnosis. α ≈ 0.48–0.65, exactly your range.
2. What a strict per-layer α fix does — measured, not asserted
We applied your strict per-layer α (not median normalization), kept everything else bit-exact, and ran a full controlled battery on the exact same stack (vLLM 0.27.1, same harness, same hardware: single RTX PRO 6000).
| Metric | official BF16 | +SSMFIX (per-layer α) | Δ |
|---|---|---|---|
| MT-Bench avg (DeepSeek-V4-Flash judge) | 7.05 | 7.47 | +0.42 |
| IFEval inst-level strict | 0.6247 | 0.6343 | +0.96pp |
| GSM8K strict (5-shot) | 0.9606 | 0.9644 | +0.38pp |
| TruthfulQA gen rouge1 | 0.2840 | 0.3452 | +6.1pp |
| TruthfulQA gen rouge2 | 0.1616 | 0.2460 | +8.4pp |
| TruthfulQA gen rougeL | 0.2803 | 0.3452 | +6.5pp |
| TruthfulQA gen bleu | 0.1775 | 0.2558 | +7.8pp |
| TruthfulQA mc1 / mc2 | 0.3647/0.5418 | 0.3758/0.5513 | +0.0111 / +0.0095 |
| CMMLU (68 subtasks) | 0.7179 | 0.6996 | −1.8pp |
9 of 11 metrics ≥ official. Generation quality and truthfulness improve substantially (TruthfulQA-gen +6~8pp, MT-Bench +0.42), and the "overthinking" pattern you documented is visibly reduced. The single notable cost: CMMLU −1.8pp (knowledge-heavy closed-book recall).
3. On "you cannot fix this by patching a few tensors"
Partial refutation, with nuance. A small tensor patch does move the model meaningfully — our data shows it is not a free lunch, but it is also not a write-off:
- Generative metrics moved large (TruthfulQA-gen +6~8pp) — the drift was real and the fix effective.
- But CMMLU −1.8pp confirms your broader point that the repair trades knowledge capacity for generation quality; the pathway damage is not simply reversible.
So: your diagnosis is confirmed, your pessimism is partially justified, and the measured verdict is "worth it for generative use, not for knowledge-max use".
4. Release (public, weights + full eval + method)
redashes/Qwen3.8-27B-BF16-SSMFIX — now public on Hugging Face, with bilingual README, full eval tables, fix report, and method credit to you and FGDumitru (qwen-ssm-repair, who independently re-implemented this and got overlapping α 0.61–0.70):
https://huggingface.co/redashes/Qwen3.8-27B-BF16-SSMFIX
Thanks again for the original investigation — it was the right lead.
Independent verification of your conv1d diagnosis — full benchmark results
Thank you for this write-up — it prompted us to run an independent, controlled verification, and your diagnosis holds up. We are the hermes-nova team (org
redashes), working with officialQwen/Qwen3.8-27BBF16 weights.1. Diagnosis confirmed by independent measurement
We measured
linear_attn.conv1d.weightstd across the 48 GatedDeltaNet layers on the official BF16 weights, and flagged the same 8 layers using astd/median > 1.6rule:
Layer our α your α match 52 0.59005 0.59005 ✓ 53 0.55484 0.55484 ✓ 56 0.54486 0.54486 ✓ 57 0.53574 0.53574 ✓ 58 0.60972 0.60972 ✓ 60 0.48136 0.48136 ✓ 61 0.65327 0.65327 ✓ 62 0.61856 0.61856 ✓ Layer set and α values are identical — two independent implementations, convergent diagnosis. α ≈ 0.48–0.65, exactly your range.
2. What a strict per-layer α fix does — measured, not asserted
We applied your strict per-layer α (not median normalization), kept everything else bit-exact, and ran a full controlled battery on the exact same stack (vLLM 0.27.1, same harness, same hardware: single RTX PRO 6000).
Metric official BF16 +SSMFIX (per-layer α) Δ MT-Bench avg (DeepSeek-V4-Flash judge) 7.05 7.47 +0.42 IFEval inst-level strict 0.6247 0.6343 +0.96pp GSM8K strict (5-shot) 0.9606 0.9644 +0.38pp TruthfulQA gen rouge1 0.2840 0.3452 +6.1pp TruthfulQA gen rouge2 0.1616 0.2460 +8.4pp TruthfulQA gen rougeL 0.2803 0.3452 +6.5pp TruthfulQA gen bleu 0.1775 0.2558 +7.8pp TruthfulQA mc1 / mc2 0.3647/0.5418 0.3758/0.5513 +0.0111 / +0.0095 CMMLU (68 subtasks) 0.7179 0.6996 −1.8pp 9 of 11 metrics ≥ official. Generation quality and truthfulness improve substantially (TruthfulQA-gen +6~8pp, MT-Bench +0.42), and the "overthinking" pattern you documented is visibly reduced. The single notable cost: CMMLU −1.8pp (knowledge-heavy closed-book recall).
3. On "you cannot fix this by patching a few tensors"
Partial refutation, with nuance. A small tensor patch does move the model meaningfully — our data shows it is not a free lunch, but it is also not a write-off:
- Generative metrics moved large (TruthfulQA-gen +6~8pp) — the drift was real and the fix effective.
- But CMMLU −1.8pp confirms your broader point that the repair trades knowledge capacity for generation quality; the pathway damage is not simply reversible.
So: your diagnosis is confirmed, your pessimism is partially justified, and the measured verdict is "worth it for generative use, not for knowledge-max use".
4. Release (public, weights + full eval + method)
redashes/Qwen3.8-27B-BF16-SSMFIX— now public on Hugging Face, with bilingual README, full eval tables, fix report, and method credit to you and FGDumitru (qwen-ssm-repair, who independently re-implemented this and got overlapping α 0.61–0.70):https://huggingface.co/redashes/Qwen3.8-27B-BF16-SSMFIX
Thanks again for the original investigation — it was the right lead.
Thank you very much for sharing testing results with SSM fix. This is the right path to take open source. We should not blindly use models shared by companies. A neural network is not a black box; it is a statistical machine governed by fundamental mathematical laws. Let's hope Alibaba Group will fix this issue in Qwen4.0 during training process.
Just curious, I was testing qwen3.8-27b and found out the same issue about the abnormal phenomenal in last 8 layer. But I'm new to AI model, not really sure what actually happens here and how this could affect the model. Can someone help me walk through this. Lots of thanks !
Just curious, I was testing qwen3.8-27b and found out the same issue about the abnormal phenomenal in last 8 layer. But I'm new to AI model, not really sure what actually happens here and how this could affect the model. Can someone help me walk through this. Lots of thanks !
Just vist this page: https://huggingface.co/redashes/Qwen3.8-27B-BF16-SSMFIX . It will explain everything.
Also here gguf: https://huggingface.co/Luis23333/Qwen3.8-27B-SSMFIX-UD-Q3_K_XL-GGUF
I looked up Qwen's thesis about their architecture. Seems like those conv1d caused by its RMS norm and designed in this way. Thesis link: https://arxiv.org/abs/2601.22966
Hi @ClariS1913 — happy to walk through your question, since we independently reproduced this scan on the official BF16 weights (hermes-nova team, org redashes). Short version combining your arXiv link and our measurements:
What the numbers are: Qwen3.8-27B is a hybrid stack — mostly GatedDeltaNet (linear-attention SSM) layers, each with a short linear_attn.conv1d conditioning the signal before SSM recurrence. Weight std sits near ~0.043 in the first ~40 layers; in the last layers (52, 53, 56, 57, 58, 60, 61, 62) it is ~1.6-1.9x that — that is the anomaly in the table above.
Is it "by design"? Partially, and the two ideas fit together. The Qwen paper (arXiv:2601.22966) argues that large outlier activations coupled with normalization (attention sinks / residual sinks, gated rescaling around RMSNorm-style layers) are functional: they act as scale factors stabilizing training, not junk to clip. On that reading the inflated late-layer scales are intentional signal conditioning, so the official weights are not broken architecturally. Note the paper studies activation outliers and normalization, though — it does not claim the conv1d weight-std spread itself is an optimum. Our controlled battery suggests the practical effect is a trade-off: strict per-layer alpha rebalancing on exactly these 8 tensors (everything else bit-exact) yields large generative gains (TruthfulQA-gen +6~8pp, MT-Bench up, visibly less overthinking) at a small knowledge cost (CMMLU -1.8pp). "Designed" and "worth rebalancing for your workload" are not contradictory: generation-heavy use benefits from the repair; knowledge-recall-heavy use can stay on official weights.
How it affects behavior: inflated conv1d scale drives the recurrent state toward saturation earlier in decoding. Short prompts look fine (perplexity normal), which is why it hides; at long context (75k+) the state drifts — answers that do not stop at the right time, repetition loops, mid-generation truncation, explanatory drift. Rescaling toward the peer median restores headroom in the recurrence, which is the mechanism behind the score jumps.
Full per-layer table and eval: https://huggingface.co/redashes/Qwen3.8-27B-BF16-SSMFIX