Qwen3.8 Overthinks :(
Thanks for this variant!
Can you suggest some settings so we can cut this overthinking part of this model which is insane, overthinking is eating most of the tokens and wasting time?
Any suggestion?
Thanks!
it would help to know what engine you're running. On Ollama use "think:medium"
@that80-20 :I am using Medium variant of the model with LM Studio and Opencode. In opencode, I don't have such setting though I have a token budget for thinking which I have never used before as I am not sure of suitable tokens for the thinking. Qwen3.6 27B that I also have, also thinks a lot but Qwen3.8 is thinking like 5-10 times more.
Use the "low" gguf variant. I tried both medium and low, but medium overthinks a lot more. Low is more concise and produces the same result in less tokens (at least I didn't notice any performance degradation). Plus it's faster.
I've been noticing that as I use that as my daily driver too. Unfortunately it seems to be widespread for Qwen3.8 indeed: https://huggingface.co/Qwen/Qwen3.8-27B/discussions/76
I haven't tried hard to mitigate that yet. What I'm doing with OpenCode is using a non-thinking configuration as a variant for simpler tasks. The model is smart enough to tackle them without requiring a reload, and speeding up those tasks significantly.
I've been able to test a bit with opencode. The regular variant support didn't work for me, and the model was reasoning so hard for simple things that that was unbearable. I needed to inject the right metadata for that to work. I tested then with low reasoning. I think it still went a bit overboard, but way better than it did minutes before for the same prompt. Then I changed the default variant for me to be non-thinking.
I use mudler's LocalAI in front of llamacpp, so I'm not 100% sure if the metadata approach will work for you. Though that's pretty standard nowadays, and directly supported by the mode - so as long as LM Studio passes it along, that should work.
This is the relevant part of my opencode config,if that helps:
"Qwen3.8-27B-NVFP4-MTP-LOW": {
"name": "Qwen3.8-27B-NVFP4-MTP-LOW",
"reasoning": false,
"metadata": { "enable_thinking": "false" },
"limit": {
"context": 307200,
"output": 32768,
"input": 274432
},
"variants": {
"nonthinking": {
"reasoning": false,
"metadata": { "enable_thinking": "false" }
},
"low": {
"reasoning": true,
"metadata": { "enable_thinking": "true", "reasoning_effort": "low" }
},
"medium": {
"reasoning": true,
"metadata": { "enable_thinking": "true", "reasoning_effort": "medium" }
},
"high": {
"reasoning": true,
"metadata": { "enable_thinking": "true", "reasoning_effort": "high" }
},
"xhigh": {
"reasoning": true,
"metadata": { "enable_thinking": "true", "reasoning_effort": "xhigh" }
}
}
}
I have modified the chat template and set the default reasoning_effort to medium, tested and it still had same issue. Then I set it to low, it still overthink.
Another issue I found is that, it steer its thinking to other files which are not asked for and waste lot of time to think and compare the content of the file in question with other files which are not related or asked for.
I never had such issue with Qwen3.6 but this, I think, Qwen3.8 is impractical model to work with, wasting tokens, time and electricity. It does lots of combination and permutation of the expected results during thinking and planning.
Horrible experience with it, at least, for codebase refactoring task. Lots of expectation but it is a dud for me. I'm going back to Qwen3.6.
Thank you very much for your efforts though!
@kremerneil No problem. Sorry to hear that, though. I've just started trying out actual coding sessions with it yesterday, so I'm still getting a taste of how it behaves on that front.
On a related note, I was using as my daily driver the amazing DavidAU's Fable Fusion 711 Qwen3.6 build. I made my own NVFP4-hybrid versions of it same way as I did for Qwen3.8, with nice results. In case you'd like to try that out: https://huggingface.co/esatapedico/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-NVFP4-GGUF
@esatapedico are you maybe planning to experiment with NVFP4 version of https://huggingface.co/redashes/Qwen3.8-27B-BF16-SSMFIX ?
I didn't know about that one. It looks really interesting, and a great job from redashes. I have been using community-made nvfp4 checkpoints because I don't generally have enough VRAM to convert them to nvfp4 myself. I'm looking into some low-mem modes, but I haven't tested yet, and it could take some time. I'll try that out as soon as I can.
I see there that they're preparing a follow-up version, so perhaps I'll aim for converting that one: https://huggingface.co/redashes/Qwen3.8-27B-BF16-SSMFIX/discussions/1#6a846662e077188b28eb6da9
I didn't know about that one. It looks really interesting, and a great job from redashes. I have been using community-made nvfp4 checkpoints because I don't generally have enough VRAM to convert them to nvfp4 myself. I'm looking into some low-mem modes, but I haven't tested yet, and it could take some time. I'll try that out as soon as I can.
I see there that they're preparing a follow-up version, so perhaps I'll aim for converting that one: https://huggingface.co/redashes/Qwen3.8-27B-BF16-SSMFIX/discussions/1#6a846662e077188b28eb6da9
I am more than happy to help out but I do not have exp in that regard, so I would need to get some guidance on execution if that would be okay.
Thank you, @BrX19 ! I've investigated, and that should be doable with my hardware in a few hours. But for larger conversions coming our way I might reach out to you for help if you're still willing then 😁
About the SSMFIX, I found it interesting enough that's worth an experiment. So I'm uploading right now all my tiers with the fix to https://huggingface.co/esatapedico/Qwen3.8-27B-NVFP4-MTP-SSMFIX-GGUF, together with a few (mixed) observations.
The more people would like to try that out, the better insights we'll have, so I hope that my upload helps with that.