jialinyyzz (Stephen Yu)

replied to their post 3 days ago

Ok, ran the binning test. Basically, no, it's not a near-tie thing. Chinese really was more poorly quantized (in 2 bit).

In BF16, it's almost the same difficulty level for both languages. Output tokens have 18.8% (EN) 20.4% (ZH) near-ties (p1-p2 < .2). So it's hard to use the mix as an explanation.

In raw IQ2_XS, chinese kl is 1.3-6.4x english for all p1 bins. Most of the difference (94%) is within the bin. The differences are largest in places where bf16 is most confident (p1 >= .99) - chinese top1 flips 6.2% (vs 0.4% for english). This is the reverse of what you would expect for the near-tie thing.

In Q4_K_M, Q6_K, and new 3.9 GB 2-bit, the curves are roughly aligned in low/mid bins. The remaining difference is about 8% and primarily in p1 >= .9 bins (very low absolute kl). 2-bit reduces output token kl from .315/.573 (EN/ZH) to .056/.060.

So yes, raw low-bit quant hurts chinese more. But it gets fixed by all the other steps (mixed precision, code training, scale distill). It didn't get fixed by just using a more chinese imatrix. Thanks for the suggestion! That definitely cleared things up quite a bit.

replied to their post 3 days ago

Nice read, thanks for the detailed analysis. For the distilled set it was about 72/28 for EN/ZH: 25.9k EN/10.2k ZH docs, 12.6M/4.8M tokens, so the minority ZH picked up more.

After this I have a result that partially confirms and partially complicates this: Training of the codes themselves (layer wise QAT, leaving scales to next stage) also favoured chinese more on Q2_K: ZH .483 --> .259 vs EN .354 --> .238, so ZH/EN changed from 1.36 to 1.09. So there is part of the gap also in codes, not just scales. Adding mixed precision + codes training + scale distillation now, for 3.9 GB build I'm at .106 / .106, so that's when the gap completely disappears.

I haven't checked the binning yet. It's a great idea and cheap to do. Will post curves once I have them.

reacted to their post with 🔥 3 days ago

Post

119

Most "AI humanizers" are tuned against a detector. This one never saw one.

humanizer-gemma-4-e4b rewrites an AI draft so it reads like a person wrote it — while keeping every fact, number, name and date. The reward during RL was an LLM judge scoring fidelity against an atomic fact list of the draft, plus a penalty for reusing the draft's phrasing and syntax. No detector anywhere in the loop, by design: optimise against a classifier and you learn that classifier, not writing.

On a 39-case everyday-writing set (62 English samples, two independent judges, an error counts only if both report it):

- 0 / 62 critical fidelity errors (v1, SFT+DPO: 3 / 62)
- 0 samples copying > 35 % of the draft's 5-grams (baseline 4B rewriter: 33 / 93)
- reuse median 0.29 — the model rewrites, it does not shuffle
- Originality.ai rates 85 % of outputs "human" — measured once, after the fact, never optimised

Two things worth knowing before you try it: it is a base-model completion, not a chat model (the exact instruction wrapper ships in prompt_format.json — reproduce it byte for byte, and turn off Ollama/LM Studio's chat template), and it drops a qualifier in roughly 1 in 3 outputs ("an estimated 4.2 %" becomes "4.2 %"). Proofread numbers and the direction of every claim.

bf16 + GGUF (Q8_0 / Q6_K / bf16) in one repo. Q5 and below are withheld — fidelity collapses.

Try it: jialinyyzz/humanizer
Model: https://huggingface.co/jialinyyzz/humanizer-gemma-4-e4b
How it was trained: https://huggingface.co/blog/jialinyyzz/we-built-an-ai-humanizer-and-never-let-it-see-a-de
Code and eval set: https://github.com/sgaofen/humanizer

reacted to their post with 🔥 3 days ago

Post

4850

For a single-purpose 12B rewriter, how much loss per bit? Chinese breaks first.

We used llama.cpp's --kl-divergence for jialinyyzz/humanizer against bf16 weights, on our held out eval (drafts+rewrites) for English and Chinese. Standard llama-quantize (but our imatrix) without additional training. "differ" means top-choice token is not the same as what bf16 selects.

- Q6_K, 10.0 GB: KL .0030 EN / .0027 ZH. About 2 in 100 tokens differ in both.
- Q4_K_M, 7.6 GB: .0198 / .0204. About 6 in 100.
- IQ3_XXS, 4.7 GB: .138 / .175. 14 vs. 17 in 100.
- IQ2_XS, 3.8 GB: .487 / .762. 26 vs. 35 in 100.

Both languages track when you go as low as 4 bits. Below that Chinese drops off faster (Chinese KL is 1.3x English at 3 bits and 1.6x at 2 bits). Not clear why. One clue: when we added more Chinese to an imatrix (1/3 of the 800k tokens), it reduced Chinese KL by 4.7% at 2 bits. English didn't change.

(The ruler: These were running at about 8k tokens for each language. A 30690 token run agrees with English, but suggests Chinese was ~10% undercounted for 4 and 3 bits. So a bit more difference)

WIP/not released. We're distilling only the fp parts of the GGUF (block scales and norms) vs. bf16. Freezing integer codes. At 2 bits we're seeing approx. 1/2 KL (so far .487 -> .263 EN, .762 -> .319 ZH; different tokens 26 -> 20, 35 -> 22 / 100). Barely changed at 4 bits so we stopped there. No new stuff to grab yet.

4 replies

·

posted an update 4 days ago

Post

4850

For a single-purpose 12B rewriter, how much loss per bit? Chinese breaks first.

We used llama.cpp's --kl-divergence for jialinyyzz/humanizer against bf16 weights, on our held out eval (drafts+rewrites) for English and Chinese. Standard llama-quantize (but our imatrix) without additional training. "differ" means top-choice token is not the same as what bf16 selects.

- Q6_K, 10.0 GB: KL .0030 EN / .0027 ZH. About 2 in 100 tokens differ in both.
- Q4_K_M, 7.6 GB: .0198 / .0204. About 6 in 100.
- IQ3_XXS, 4.7 GB: .138 / .175. 14 vs. 17 in 100.
- IQ2_XS, 3.8 GB: .487 / .762. 26 vs. 35 in 100.

Both languages track when you go as low as 4 bits. Below that Chinese drops off faster (Chinese KL is 1.3x English at 3 bits and 1.6x at 2 bits). Not clear why. One clue: when we added more Chinese to an imatrix (1/3 of the 800k tokens), it reduced Chinese KL by 4.7% at 2 bits. English didn't change.

(The ruler: These were running at about 8k tokens for each language. A 30690 token run agrees with English, but suggests Chinese was ~10% undercounted for 4 and 3 bits. So a bit more difference)

WIP/not released. We're distilling only the fp parts of the GGUF (block scales and norms) vs. bf16. Freezing integer codes. At 2 bits we're seeing approx. 1/2 KL (so far .487 -> .263 EN, .762 -> .319 ZH; different tokens 26 -> 20, 35 -> 22 / 100). Barely changed at 4 bits so we stopped there. No new stuff to grab yet.

4 replies

·

posted an update 22 days ago

Post

119

Most "AI humanizers" are tuned against a detector. This one never saw one.

humanizer-gemma-4-e4b rewrites an AI draft so it reads like a person wrote it — while keeping every fact, number, name and date. The reward during RL was an LLM judge scoring fidelity against an atomic fact list of the draft, plus a penalty for reusing the draft's phrasing and syntax. No detector anywhere in the loop, by design: optimise against a classifier and you learn that classifier, not writing.

On a 39-case everyday-writing set (62 English samples, two independent judges, an error counts only if both report it):

- 0 / 62 critical fidelity errors (v1, SFT+DPO: 3 / 62)
- 0 samples copying > 35 % of the draft's 5-grams (baseline 4B rewriter: 33 / 93)
- reuse median 0.29 — the model rewrites, it does not shuffle
- Originality.ai rates 85 % of outputs "human" — measured once, after the fact, never optimised

Two things worth knowing before you try it: it is a base-model completion, not a chat model (the exact instruction wrapper ships in prompt_format.json — reproduce it byte for byte, and turn off Ollama/LM Studio's chat template), and it drops a qualifier in roughly 1 in 3 outputs ("an estimated 4.2 %" becomes "4.2 %"). Proofread numbers and the direction of every claim.

bf16 + GGUF (Q8_0 / Q6_K / bf16) in one repo. Q5 and below are withheld — fidelity collapses.

Try it: jialinyyzz/humanizer
Model: https://huggingface.co/jialinyyzz/humanizer-gemma-4-e4b
How it was trained: https://huggingface.co/blog/jialinyyzz/we-built-an-ai-humanizer-and-never-let-it-see-a-de
Code and eval set: https://github.com/sgaofen/humanizer

Stephen Yu PRO

AI & ML interests

Recent Activity

Organizations

Stephen Yu PRO

AI & ML interests

Recent Activity

Organizations

jialinyyzz's activity