There is no benchmark for mlx models
It's clear about UD GGUF, but almost nothing about MLX Dynamic Quantization. How good is that versus basic 4-bit by MLX-Community? There is a useful image in the section Unsloth GGUF Benchmarks; it would be good to put MLX versions on that graph.
Hi.
I've compared KLD of oQ, Q, MXFP and UD MLX quantizations, see detailed results.
Thank you! Can you provide the link for MXFP4 that you used in your test? On GitHub, in detailed results, mxfp4 are 17.16 GB, but the one that I use is 19 GB (OsaurusAI/Qwen3.6-35B-A3B-mxfp4). I wonder, why is there a difference?
I haven't uploaded it, but it's a simple mlx_vlm.convert --hf-path ~/.cache/huggingface/models/Qwen/Qwen3.6-35B-A3B --mlx-path ./Qwen3.6-35B-A3B-MLX-VL-MXFP4 -q --q-mode mxfp4 one.
The https://huggingface.co/OsaurusAI/Qwen3.6-35B-A3B-mxfp4 you have was quantized with explicit --q-group-size 32, while the default for MXFP4 is 64. Smaller group size leads to higher accuracy but requires more RAM.
Thank you for sharing. Really useful to compare, especially for lower end hardware.
Curious if you could compare the oQ3.5e and oQ4e variants as well. Couldn't really find a benchmark ANYWHERE of these "enhanced" oQ's
Curious if you could compare the oQ3.5e and oQ4e variants as well. Couldn't really find a benchmark ANYWHERE of these "enhanced" oQ's
Let's move there https://github.com/jundot/omlx/discussions/1019 :)
I've been following the KLD benchmarks, so I just tested the models in omlx. The default 4bit quant I made locally with mlx_vlm.convert scored better in livecodebench and MBPP compared to this model. The weights are more optimized for UD, but it's slower to run and I don't know why it would score worse? Are my settings off? Or is there something about the model weights we don't fully understand yet.
Also relevant, when I locally converted to 6 bits, it also performed much worse on all the benchmarks. The KLD clearly shows it's closer to the main model, but why would the benchmarks get worse for UD and 6 bit?
The default 4bit quant I made locally with mlx_vlm.convert scored better in livecodebench and MBPP compared to this model.
I'd ask you to provide exact commands you've used to quantize. Regardless of a potential incorrectness of my KLD test suite, affine Q4 should not perform better than a smart dynamic quantization.