Upload README.md with huggingface_hub
Browse files
README.md
ADDED
|
@@ -0,0 +1,25 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# GoLLeM-v5 — modele (backup lokalny)
|
| 2 |
+
|
| 3 |
+
Backup ckptów treningowych GoLLeM-v5 (scan tokenów 16M). Źródło: RunPod RTX 5090.
|
| 4 |
+
Redundancja: pod (oryginał) + HF `SlayerLab/minimal-en-corpus-5b:ckpts/` (durable) + ten katalog (lokalny).
|
| 5 |
+
|
| 6 |
+
## Modele (scan tokenów, single-factor, BPE-12k, L6/d408/h6, block1024, batch64, AdamW lr6e-4→6e-5, seed1337)
|
| 7 |
+
|
| 8 |
+
| plik | tokeny | epoki | BLiMP | ARC-Easy | WikiText-2 BPB | board-pozycja |
|
| 9 |
+
|------|--------|-------|-------|----------|----------------|---------------|
|
| 10 |
+
| run_bpe16m_c_ckpt.pt | 3.2B | 1.19 | 71.54 | 39.39 | 1.2161 | #16/74 |
|
| 11 |
+
| run_bpe16m_6b_d_ckpt.pt | 6B | 2.2 | 73.29 | 41.04 | 1.1943 | #11/74 |
|
| 12 |
+
| run_bpe16m_10b_e_ckpt.pt| 10B | 3.7 | 73.43 | 41.67 | 1.1815 | (liczy Latarnik) |
|
| 13 |
+
|
| 14 |
+
**Wniosek scanu:** BLiMP saturuje ~6B na 16M (Δ6→10=+0.14); ARC dalej rośnie @10B (knee>10B); BPB monotoniczny.
|
| 15 |
+
|
| 16 |
+
- `tokenizer.json` — BPE-12k (vocab 12288), ZALOCKOWANY (werdykt 3/3 vs byte-baseline: BLiMP+2σ ∧ BPB-Δ0.238 ∧ board-#16). Wspólny dla wszystkich modeli v5.
|
| 17 |
+
- Config eval: `byte_lm_eval_bpe.py --tokenizer tokenizer.json --ckpt <model> --tasks wikitext,blimp,arc_easy --n-head 6`.
|
| 18 |
+
- Korpus treningowy: `SlayerLab/minimal-en-corpus-5b` (2.70B unique BPE-tok, FineWeb-Edu, ARC-targeted, decontam'd).
|
| 19 |
+
|
| 20 |
+
## Kolejne (dojdą po treningu)
|
| 21 |
+
- 32M-baseline (L6/d576/h9, izolacja-rozmiaru) + lewary single-factor (RoPE/low-rank/Muon/depth).
|
| 22 |
+
- 150M flagowiec (+KARD-distill z Qwena).
|
| 23 |
+
|
| 24 |
+
Prowieniencja pełna: labvault `21_09_GoLLeM-v5-Skalowanie-Glint/` (Stan.md + artifacts + S2-Wnioski).
|
| 25 |
+
Data backupu: 2026-09-22.
|