F16 or BF16?
When I look at the linear attention metadata it says that it's F16, not BF16. Which is it?
I'm also very curious about this question.
Yes, it is actually in FP16, as the model was loaded in FP16 during quantization for better precision.
Thank you for your reply! I'm a bit confused about one detail: since the original Qwen 3.6 27B model is in BF16, my understanding is that keeping it in BF16 would maintain the highest precision. Could you explain why FP16 was used instead? Thank you very much!
Yes, loading the model in BF16 would correctly preserve the weight values from the model providers.
But in calibration and quantization, FP16 higher precision provides better accuracy compared to BF16, and this compensate for the FP16 loss in weight loading. Based on the current cyankiwi AWQ models, calibrating and quantizing models in FP16 provide a few % lower KL divergence than in BF16.
For instance, 0.1 is represented as 0.0999755859 in FP16 and 0.1000976562 in BF16, which FP16 is ~4× more accurate here.
Thank you so much for the response! I'm pretty new to this space, so apologies if this is a dumb question. I originally thought that something like the approach in https://huggingface.co/Qwen/Qwen3.5-27B-GPTQ-Int4 — keeping non-quantized params in BF16 + AWQ (I know the official one uses GPTQ, just using it as an example of the format idea) — would yield the highest precision. But from what you're saying, it seems like FP16 might actually be the better choice?
I'm still a bit puzzled though — BF16 and FP16 have different exponent and mantissa bit allocations. Wouldn't converting from BF16 to FP16, then running quantization calibration and saving in that format, introduce some precision loss?
Ah, I forgot to update this — I think I’ve found the answer now.
I was originally worried that converting the original BF16 weights to FP16 might cause range-related loss or overflow. But I checked the original Qwen3.6 27B weights, and the maximum absolute value was only 25.5, far below FP16’s limit of 65504.
I wrote a bit more about the check here:
https://x.com/0xkeenz/status/2074970081050906752
So my concern about BF16 → FP16 causing overflow was unnecessary. For these weights, FP16 fits safely, and its finer mantissa can be more helpful during calibration/quantization.
Thanks again, and thanks to cpatonn for the earlier explanation!