FIT-13.5G — max quality on linux amd 9070xt
I typically start with the highest quality i can run for my pc, then test, re-test and the FIT 13.5 showed great speed improvements over the Q4_K version that i downloaded first from another high quality quantifying group. The FIT ver. has doubled my t/s and produced top notch reasoning for what i asked it. I have only so far prompted it to look at some known buggy scripts. They are difficult benchmarks for models to pass in first try and the FIT 13.5 found the bugs easy but also suggested the patch's to use modern linux dependencies already installed and that is part of the test.
The only issues i saw was when i didn't use any sampling settings and the model just ran out of ctx to answer. max thinking mode was engaged .
I was able to get double the t/s from 14+ up to 30+ and for this model and on my pc is a huge win. Also i wanted to try the new FIT version. This is very neat quantization project and that's why im leaving some feedback.. thanks Scorpio117
I think i will DL the FIT 12 next, not that i expect vast t/s increases but having more room for ctx is always nice.
here is my current coder mode launcher
/llama.cpp/build/bin/llama-server \
-m /Qwen3.8-27B-Uncensored-FIT-13.5G-IQ4_XS.gguf \
-ngl 99 -c 19999 -t 6 \
--temp 0.7 --top-p 0.80 --min-p 0.0 --top-k 20 \
--reasoning-preserve -fa on \
--repeat-penalty 1.0 --presence-penalty 1.5 \
--host 127.0.0.1 --port 9931
4281 t, 2m 20s, 30.50 t/s
i will pick quality over quantity every time ,
and it's awesome running the FIT 13.5 qwen3.8-27b on a 16gb card 😎
- also i have used the yarn flags as well but this needs more testing
--rope-scaling yarn --yarn-orig-ctx 8192 --rope-freq-scale 0.125 \
i got this sampling from the qwen main model page. not sure if it helps but that's why i test a lot :]
https://huggingface.co/Qwen/Qwen3.8-27B
We recommend using the following sets of sampling parameters for generation:
Thinking Mode: temperature=1.0, top_p=0.95, top_k=20, min_p=0.0, presence_penalty=0.0, repetition_penalty=1.0
Instruct (or non-thinking) mode: temperature=0.7, top_p=0.80, top_k=20, min_p=0.0, presence_penalty=1.5, repetition_penalty=1.0
Please note that the support for sampling parameters varies according to inference frameworks.
Thanks a lot for the detailed feedback! 😎
30+ t/s on a 9070 XT 16GB with the 13.5G build is really nice to see.
The ~2x speedup compared with the Q4_K quant is probably mainly because FIT-13.5G leaves enough VRAM headroom for the model + context/KV to stay comfortably on the GPU. Once a larger quant starts putting pressure on the 16GB limit or causes partial offloading, performance can drop very quickly.
And yes, your expectation for FIT-12G is pretty much what I would expect too: probably not a huge raw t/s increase, but noticeably more room for context/KV, which can be more useful in practice.
Great to hear that the coding/reasoning quality is holding up as well — that's exactly the balance FIT is trying to target: use the available memory budget as efficiently as possible instead of being limited to a few fixed quant sizes.
Thanks again for testing it on real hardware and sharing the numbers. Feedback like this is extremely useful for the project 🙂
Thanks a lot for the detailed feedback! 😎
30+ t/s on a 9070 XT 16GB with the 13.5G build is really nice to see.
The ~2x speedup compared with the Q4_K quant is probably mainly because FIT-13.5G leaves enough VRAM headroom for the model + context/KV to stay comfortably on the GPU. Once a larger quant starts putting pressure on the 16GB limit or causes partial offloading, performance can drop very quickly.
And yes, your expectation for FIT-12G is pretty much what I would expect too: probably not a huge raw t/s increase, but noticeably more room for context/KV, which can be more useful in practice.
Great to hear that the coding/reasoning quality is holding up as well — that's exactly the balance FIT is trying to target: use the available memory budget as efficiently as possible instead of being limited to a few fixed quant sizes.
Thanks again for testing it on real hardware and sharing the numbers. Feedback like this is extremely useful for the project 🙂
Is there a 35B A3B (MoE) variant available?
Thanks a lot for the detailed feedback! 😎
30+ t/s on a 9070 XT 16GB with the 13.5G build is really nice to see.
The ~2x speedup compared with the Q4_K quant is probably mainly because FIT-13.5G leaves enough VRAM headroom for the model + context/KV to stay comfortably on the GPU. Once a larger quant starts putting pressure on the 16GB limit or causes partial offloading, performance can drop very quickly.
And yes, your expectation for FIT-12G is pretty much what I would expect too: probably not a huge raw t/s increase, but noticeably more room for context/KV, which can be more useful in practice.
Great to hear that the coding/reasoning quality is holding up as well — that's exactly the balance FIT is trying to target: use the available memory budget as efficiently as possible instead of being limited to a few fixed quant sizes.
Thanks again for testing it on real hardware and sharing the numbers. Feedback like this is extremely useful for the project 🙂
Is there a 35B A3B (MoE) variant available?
Not for Qwen3.8 specifically — there isn't an official 35B-A3B Qwen3.8 model at the moment.
FIT itself does support MoE architectures though, and I've already validated it internally on 35B-A3B models. So a 35B-A3B FIT release is definitely possible. 😎
For now, I do have several 35B-A3B MoE models available with my older APEX quantization, including Ornith-1.5-35B-A3B.
FIT is still pretty new, so I'm gradually expanding it to more model families rather than rushing out a lot of builds at once.
Thanks a lot for the detailed feedback! 😎
30+ t/s on a 9070 XT 16GB with the 13.5G build is really nice to see.
The ~2x speedup compared with the Q4_K quant is probably mainly because FIT-13.5G leaves enough VRAM headroom for the model + context/KV to stay comfortably on the GPU. Once a larger quant starts putting pressure on the 16GB limit or causes partial offloading, performance can drop very quickly.
And yes, your expectation for FIT-12G is pretty much what I would expect too: probably not a huge raw t/s increase, but noticeably more room for context/KV, which can be more useful in practice.
Great to hear that the coding/reasoning quality is holding up as well — that's exactly the balance FIT is trying to target: use the available memory budget as efficiently as possible instead of being limited to a few fixed quant sizes.
Thanks again for testing it on real hardware and sharing the numbers. Feedback like this is extremely useful for the project 🙂
Is there a 35B A3B (MoE) variant available?
Not for Qwen3.8 specifically — there isn't an official 35B-A3B Qwen3.8 model at the moment.
FIT itself does support MoE architectures though, and I've already validated it internally on 35B-A3B models. So a 35B-A3B FIT release is definitely possible. 😎
For now, I do have several 35B-A3B MoE models available with my older APEX quantization, including Ornith-1.5-35B-A3B.
FIT is still pretty new, so I'm gradually expanding it to more model families rather than rushing out a lot of builds at once.
I typically use this model as the driving model for my local programming agent: https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B
"FIT itself does support MoE architectures though, and I've already validated it internally on 35B-A3B models. So a 35B-A3B FIT release is definitely possible. 😎"
Awesome, and thanks for the releases so far.
Finally sorted out a proper bench testing harness and got some legit numbers ✅
Using BenchLocal
https://github.com/stevibe/BenchLocal
BugFind-15
Temp= 1
Default Rest
Syntax & Surface
100
Logic & Algorithmic
100
Subtle & Tricky
33
Red Herring Resistance
57
Multi-Turn Debugging
100
Overall Score= 75
★★★★ Good
tg = 30.51 t/s, tg_3s = 30.82 t/s
tg = 30.55 t/s, tg_3s = 30.58 t/s
tg = 30.54 t/s, tg_3s = 30.52 t/s
To be honest, It's a solid score and only reason i think it flat missed the three was, it ran out of think ctx. All three hit 4min + and test stopped then went to next one. My next test would be simply add the suggested sampling params and send it, perhaps the --presence-penalty 1.5
would take care of the over think. 🤔
To be honest, It's a solid score and only reason i think it flat missed the three was, it ran out of think ctx. All three hit 4min + and test stopped then went to next one. My next test would be simply add the suggested sampling params and send it, perhaps the
--presence-penalty 1.5
would take care of the over think. 🤔
Finally sorted out a proper bench testing harness and got some legit numbers ✅
Using BenchLocal
https://github.com/stevibe/BenchLocal
BugFind-15
Temp= 1
Default RestSyntax & Surface
100Logic & Algorithmic
100Subtle & Tricky
33Red Herring Resistance
57Multi-Turn Debugging
100Overall Score= 75
★★★★ Goodtg = 30.51 t/s, tg_3s = 30.82 t/s
tg = 30.55 t/s, tg_3s = 30.58 t/s
tg = 30.54 t/s, tg_3s = 30.52 t/s
I think we need a locally running testing tool to evaluate the capabilities of our quantized models. The BenchLocal tool you're currently using has absolutely no global timeout settings, which leads to numerous false timeout errors during testing (e.g., timing out after just a few seconds). Our main goal is to verify correctness, rather than execution speed. While speed does matter somewhat for local models, it's not the primary focus—since most local models run relatively slowly, the excessive number of false timeouts means this tool completely fails to measure the true capabilities of local models.
4 minutes plus, means it tried for 4 minutes plus then times out , not few seconds..
4 minutes plus, means it tried for 4 minutes plus then times out , not few seconds..
What I meant was the specific cases I ran into while using this test tool, rather than your usage scenario.
I am just adding my two cents of data, and some feedback for your released model. This is the only one i have downloaded and is ongoing testing. I am still learning the whole testing side of LLM and i know the BenchLocal is new and not fully polished. I have the python codes to do a full HF mega test but at 30t/s my pc would probably not be very happy about it, hence smaller bench testing for now.
Some of my own research;
"This means your 13.5 GiB GGUF sits precisely in the target zone where mixed quantization acts as a force multiplier—saving you over 0.5 GiB of VRAM compared to IQ4_XS while keeping the KL divergence penalty drastically lower than a standard uniform IQ3_M preset"
FIT is legit way of alocating weights and i am happy to have downloaded and test this model.