add README.md

README.md

@@ -0,0 +1,66 @@
+---
+language: [ar]
+license: apache-2.0
+library_name: transformers
+pipeline_tag: fill-mask
+tags: [arabic, bert, masked-lm, encoder, nawah, tiny]
+datasets: [kaust-generative-ai/fineweb-edu-ar]
+---
+
+# Nawah-BERT-6M-v2
+
+A **5,993,600-parameter** Arabic BERT trained from scratch on **15B tokens**.
+Same architecture as [`Nawah-BERT-6M`](https://huggingface.co/oddadmix/Nawah-BERT-6M)
+(hidden 128, 8 layers, 2 heads, ctx 2,048, 32K vocab) — **only the token budget
+changed**, and it changed everything.
+
+## v1 → v2: the token budget was the whole story
+
+Held-out MLM on 1,024 chunks neither run trained on, standard 80/10/10:
+
+| | tokens | loss @15% | **ppl** | loss @30% | in-context top-1 | top-5 |
+|---|---|---|---|---|---|---|
+| v1 | 5B | 6.0424 | 420.9 | 6.1053 | 2.7% | 7.3% |
+| **v2** | **15B** | **3.3542** | **28.6** | **3.8419** | **37.2%** | **53.0%** |
+| *corpus unigram* | — | *8.031* | *3,076* | — | — | — |
+
+**Perplexity improved 14.7×** and single-mask recovery went from near-nothing to
+recovering the exact token 37% of the time.
+
+### Correcting the v1 card
+
+The v1 card said the model had *saturated* — that "5B tokens is roughly 500×
+past Chinchilla-optimal for this size, and the flat tail is what that looks
+like." **That was wrong.** The flat tail was the **cosine learning rate decaying
+to zero**, not the model running out of capacity. Warm-starting at lr 3e-4 on
+fresh tokens moved held-out loss 6.04 → 3.35. A 1.9M-parameter backbone had far
+more to give than its own loss curve suggested.
+
+## Training
+
+Continued from v1: 10B **unseen** tokens (permutation chunks 2,441,344 onward of
+`tokens_20B_sep.bin`, so no repetition of what v1 saw), lr 3e-4 cosine, 38,146
+steps at 262,144 tokens/step, mask 30%→15% annealed at step 33,949, bf16,
+`torch_compile`. 3.5 h on a single consumer GPU. Total across both runs: 15B of
+the corpus's 20B.
+
+## Downstream
+
+Fine-tuned on three tasks and compared against the Llama models built for each.
+Full numbers in each card:
+
+| task | v1 | **v2** | reference (Llama) |
+|---|---|---|---|
+| [RuleCheck](https://huggingface.co/oddadmix/Nawah-RuleCheck-BERT-6M-v2) unseen wording | 0.9357 | **0.9598** | 0.9949 (5M) |
+| [Guard](https://huggingface.co/oddadmix/Nawah-Guard-BERT-6M-v2) held-out over-refusal ↓ | 0.0146 | **0.0115** | 0.0094 (52M) |
+| [Router](https://huggingface.co/oddadmix/Nawah-Router-BERT-6M-v2) unseen category sets | 0.9137 | **0.9327** | 0.9308 (52M) |
+
+### Use mean pooling, not [CLS]
+
+Pretrained on **packed 2,048-token chunks with no `[CLS]`**, so position 0 has no
+summary role and `BertForSequenceClassification`'s pooler arrives randomly
+initialised. With it the model never leaves the class prior at any learning rate
+from 1e-4 to 3e-3. Mean-pool over non-pad positions instead — see
+`bert_meanpool.py` in the task repos.
+
+© KAND CA 2026 — PROJECT NAWAH