Benchmark validity?
I have compared Ornith 1.0 35B A3B in APEX I-Quality quantization with GRM-3.2-Sky in Q4_K_M quantization in Benchlocal with 9x runs. The same sampling parameters were used for both:
jinja = true
reasoning-preserve = true
temp = 0.6
top-p = 0.95
top-k = 20
repeat-penalty = 1.05
b = 16384
ub = 2048
HermesAgent-20:
Ornith 1.0 35B A3B: 88
GRM-3.2-Sky: 83
BugFind-15:
Ornith 1.0 35B A3B: 92
GRM-3.2-Sky: 85
Note that this is an unfair comparison since Ornith is using a superior quaint, but I don't have the hardware to quantize 35B models currently. Will update if someone releases APEX quantizations for this model. Still, the fact that GRM-3.2-Sky, which is supposedly superior to Ornith-1.0-35B according to benchmarks, is worse than it even at Q4_K_M is suspicious and brings the validity of GRM-3.2-Sky's benchmarks into question.
im making some of my own quants- its based on apex- but i modified it slightly
ill comment the link here when theyre done cooking!
Just curious, can you quantize a 35B model (~70gb) if you only gave 32GB ram?
Just curious, can you quantize a 35B model (~70gb) if you only gave 32GB ram?
you can- I did but on 12gb
just takes a painstaking amount of python orchestration- around like 2000 lines
I might open source- but im a perfectionist so idk when ill release it
I do quantize models I like with it- so just @ me and ill consider giving it a quant!
I'll post my results in a few hours. HermesAgent-20 and BugFind-15 are the main ones I care about, but if you want I can run others. I have a R9700 AI PRO, so I will run it in Quality mode. (though I use Q8 KV if possible so I might need to reduce ctx size to like 200k)
I'll post my results in a few hours. HermesAgent-20 and BugFind-15 are the main ones I care about, but if you want I can run others. I have a R9700 AI PRO, so I will run it in Quality mode. (though I use Q8 KV if possible so I might need to reduce ctx size to like 200k)
Is there the update?
el4's quaint ended up being broken, so I'm waiting until it is fixed.
working on the fix right now
my refactor broke the mapping- a fix should be either ready today or tmrw
Sampling parameters:
reasoning-preserve = true
jinja = true
temp = 0.6
top-p = 0.95
top-k = 20
repeat-penalty = 1.05
HermesAgent-20: 79 (Ornith 1.0 35B got 88)
BugFind-15: 84 (Ornith 1.0 35B got 92)
Thanks to el4 for releasing a good quality APEX quaint for this model! These are the 2 benchmarks I value the most, but I am open to running any others included in BenchLocal.
Wait, decrease?
Yup, same results as with Q4_K_M. If you want, I'm making a github repo of a bunch of models to see how they actually stack up against each other in BenchLocal. Planning to add this one too.
https://github.com/BarerRocket2678/AI-Model-Benchmarks/tree/main
How if it in BF16 format? Hopefully you can help ya, thankyou so much for the insights :D<
Yup, same results as with Q4_K_M. If you want, I'm making a github repo of a bunch of models to see how they actually stack up against each other in BenchLocal. Planning to add this one too.
https://github.com/BarerRocket2678/AI-Model-Benchmarks/tree/main
Thats expected- the goal of an apex quant is to get the same or better with lower vram usage
Thanks for running the benchs!
Unfortunately, I can't run these types of models in BF16, since that would be 70GB of VRAM. Plus, I-Quality APEX and UD-Q4_K_M have almost zero performance decrease from BF16 and at scale it is preferable to use INT4 or FP4 for increased performance. (Deepseek V4 even uses a mix of FP8 and FP4 by default)
I used Q8_0, and initially, writing code seemed quick, getting 80% of it working.
The remaining 20% involved detailed work, bug fixes, logic adjustments, and feature additions. It led me astray; fixing one thing caused another to break, haha.
I had to go back to 27B 🚀, where it handled everything in a snap.
I used Q8_0, and initially, writing code seemed quick, getting 80% of it working.
The remaining 20% involved detailed work, bug fixes, logic adjustments, and feature additions. It led me astray; fixing one thing caused another to break, haha.
I had to go back to 27B 🚀, where it handled everything in a snap.
The characteristic of this architecture is unique. I've seen, all the qwen variants are great to make the macro architecture. But, if there is the bug, we can't use the same model. Better use another moe modeltype, such gemma or OSS. They seems better at instruction following and not too much shit talking.
Check out my new quants! they use a dynamic map instead of a static apex map- so its mathematically superior to apex
From what I've seen for MoE models, they do not operate well at 4-bit quants. Also not every quantization is the same. They are very different in quality despite the same size. You should benchmark not less than Q6_K otherwise you benchmark quantization quality instead of original model quality.
Based on the APEX quantization system, they found that Q6_K is only needed for certain weights, and that the rest can be much lower. I found personally that there is almost no benchmark difference between APEX I-Quality and I-Compact, at least in agentic situations. Also, UD-Q4_K_M seems to perform just fine with MOE models, based on both my experiences and the experiences of the APEX team's results. (I can't say for any other quantizations, but I'm sure bartowski Q4_K_M is fine)
Well... I've checked out APEX methodology. And it seems that they believe that PPL is the main metric. (Correct me if I am wrong). Their own table with comparison with Unsloth shows that while APEX has better PPL, UD has better KLD. So from my understanding UD is actually better for that exact comparison. So they actually show that their method is not very good.
Why Q4_K_M is not good for benchmarking model's quality. If you look at same top p metric, it is around 95-96% (for good quants, not all Unsloth, bartowski, etc. quants are really good). So in benchmark with temp=0 you get 4-5% different tokens than original model. With temp>0 there is RMS dp metric that shows same thing but not only for top token. It is far from perfect as well. That is for good quant. For some random unmeasured quant things may be worse. So if you really want to bench model vs model quality you should bench full models. If you want to bench quant vs quant to make plausible judgement about full models you should measure each quant (not only PPL) to be sure they are on par. But with 4-bit it is not the same model as original actually. It is near, but not the same.
Well... I've checked out APEX methodology. And it seems that they believe that PPL is the main metric. (Correct me if I am wrong). Their own table with comparison with Unsloth shows that while APEX has better PPL, UD has better KLD. So from my understanding UD is actually better for that exact comparison. So they actually show that their method is not very good.
Why Q4_K_M is not good for benchmarking model's quality. If you look at same top p metric, it is around 95-96% (for good quants, not all Unsloth, bartowski, etc. quants are really good). So in benchmark with temp=0 you get 4-5% different tokens than original model. With temp>0 there is RMS dp metric that shows same thing but not only for top token. It is far from perfect as well. That is for good quant. For some random unmeasured quant things may be worse. So if you really want to bench model vs model quality you should bench full models. If you want to bench quant vs quant to make plausible judgement about full models you should measure each quant (not only PPL) to be sure they are on par. But with 4-bit it is not the same model as original actually. It is near, but not the same.
well apex isnt about keeping 100% quality- thats basically impossible even with trellis quantization
the point is achieving near q6-q8 real world use output with a q4 sized quant
From their own comparison it wins only in speed (on exact hardware it was tested). I don't see why it is better than UD they compare with. UD is better by KLD and tail KLD. So it is just another method of quantization with its own shortcomings. Not better one.
The point is as you already said: there is no way to keep 100% quality. But there are different methods and not all give the same results. I haven't seen major providers measuring all their quants for all models. Actually they don't know what they provide (otherwise I would expect they post metrics). Yes, it gives somewhat good results most of the time. But it shouldn't be taken as granted that every quant from popular provider is really good.
You may check my investigations for couple of models here (not MoE though):
https://huggingface.co/NikiKrutan/GRM-2.6-Plus-0628-MTP-GGUF
https://huggingface.co/NikiKrutan/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-GGUF
And, yeah, bartowski's Q4_K_M is really good on that exact model.
well apex quants use static maps that assume the edges need to be protected
so performance would tank on a fine tune- since different experts are impacted
thats why the results are hit or miss