AesSedai

/

Qwen3.8-Flash-Next-GGUF

like 16

There's some prompt processing penalty though. I've done some exploration in this area: https://github.com/ggml-org/llama.cpp/discussions/27864

I ran several tests with IQ4_NL quantized PLE, and it seems fine with the llama.cpp correctness fixes here. Here's a report here: https://github.com/ggml-org/llama.cpp/discussions/27965

Q8_0 for the larger quants is really close in accuracy to the BF16, I was going to try some tests on Q4_K quanting the PLE tensors this weekend actually. I think that it should still show up in PPL/KLD.

IIRC, neither the DOWN tensors nor the PLE could use K quants (they weren't multiple of 256). Had to choose between iq4_nl, q4_0, q5_0, q5_1 or q8_0. @aessedai did you check the quantization logs? it is possible that all down/ple were upcasted to q8_0 already

Just checked through the metadata viewer and the PLEs for all of my quants are Q8_0 lol.

I'm interested in seeing how much the perplexity/kld gets worse if you drop the PLE to IQ4_NL

Just checked through the metadata viewer and the PLEs for all of my quants are Q8_0 lol.

So is ffn down

Q8_0 for the larger quants is really close in accuracy to the BF16, I was going to try some tests on Q4_K quanting the PLE tensors this weekend actually. I think that it should still show up in PPL/KLD.

Did you happen to run these experiments? Really curious to hear your results! They could also tell us if it could potentially be worth using bf16 for the ngram tensors.

@Mushoz I've uploaded three new quants with Q4_0 for the PLE tensors, from a PPL / KLD perspective they look to perform better for the size but I haven't run benchmarks on them for instance. Figured they were worth putting up for people who want to shave a bit of size off.

Just checked through the metadata viewer and the PLEs for all of my quants are Q8_0 lol.

So is ffn down

I'm not yet knowledgeable enough to know if this is a good or a bad thing, re the ffn down being at Q8_0, could someone enlighten?

Just checked through the metadata viewer and the PLEs for all of my quants are Q8_0 lol.

So is ffn down

I'm not yet knowledgeable enough to know if this is a good or a bad thing, re the ffn down being at Q8_0, could someone enlighten?

If it still fits in your (v)ram it can be a good thing, as quality will be higher. But these were meant to be in Q6_K which is a smaller quant (so file size would be smaller).

If it still fits in your (v)ram it can be a good thing, as quality will be higher. But these were meant to be in Q6_K which is a smaller quant (so file size would be smaller).

OK that makes sense, I read that they were accidentally 'upscaled' and I figured that means they are at the higher quant than originally intended, but wasn't sure if that could have some adverse affect. Thanks for the info.

If it still fits in your (v)ram it can be a good thing, as quality will be higher. But these were meant to be in Q6_K which is a smaller quant (so file size would be smaller).

OK that makes sense, I read that they were accidentally 'upscaled' and I figured that means they are at the higher quant than originally intended, but wasn't sure if that could have some adverse affect. Thanks for the info.

They weren't upscaled, as that implies they were smaller to begin with. They were simply quantized less than intended, as the K quants are not compatible with the tensor shape of these particular tensor (no multiple of 256 in its dimension).

OK that makes sense, I read that they were accidentally 'upscaled' and I figured that means they are at the higher quant than originally intended, but wasn't sure if that could have some adverse affect. Thanks for the info.

They weren't upscaled, as that implies they were smaller to begin with. They were simply quantized less than intended, as the K quants are not compatible with the tensor shape of these particular tensor (no multiple of 256 in its dimension).

Right, that makes much more sense, I don't know where I got the impression of upscaling from, I am probably mixing up something I read elsewhere obviously. Thanks for taking the time to explain.

@AesSedai I've been doing some perplexity measurements and it seems like IQ4_NL on the PLE provides better results while being a bit smaller.