pageman/nanochat-filipino-p1-fixed-d20-3x

Model card — p1-fixed-d20-3x

Paper: https://www.researchgate.net/publication/412302216_Equal-Exposure_Depth_and_Held-Out_Tagalog_Bits-per-Byte_on_WikiText-TL-39
Code: https://github.com/pageman/nanochat-filipino

Intended use: research base language model for Tagalog Wikipedia-style text. Primary reported number is held-out bits-per-byte on WikiText-TL-39 (reconstructed_article_70_15_15).

Out of scope: chat, instruction following, safety filtering, official government or medical use, English CORE comparison, classification.

Data: public WikiText-TL-39 (linkanjarad/Wikitext-TL39, parquet SHA-256 706d7064…). Canonical text is source text with LF line endings only. Tokenizer is a 32,768 BPE trained on train documents only.

Training: nanochat commit 92d63d4e8bb4df75c3b71618f31ddde2378b2bcd, depth 20, T=2048, D_actual=19,267,584 tokens, Runpod A40, about 0.20 GPU-hours for this depth. No SFT.

Eval: registered primary is val_bpb_full=1.172248 from official full-split evaluate_bpb. One test_bpb=1.164768 after validation-only selection. One seed. Margin to depth 8 is 0.0069 BPB and is not interpreted as a ranking. Machine-readable bundles: https://github.com/pageman/nanochat-filipino/tree/main/results

Not the primary metric: val_bpb inside meta_000294.json (d20 1.117213) used --eval-tokens=262144 during training. Do not rank depths from that loop slice. Mid-run d12 min 1.084991 was not selected.

Carbon / compute: one A40 at $0.44/hr; the four-depth seed-0 series was under 0.5 GPU-hours of training. Extra seeds and pilots are separate.

Citations: Pajo (2026), https://www.researchgate.net/publication/412302216_Equal-Exposure_Depth_and_Held-Out_Tagalog_Bits-per-Byte_on_WikiText-TL-39; Cruz & Cheng (2019), arXiv:1907.00409; Karpathy nanochat; AsPredicted #306780. Study repository: https://github.com/pageman/nanochat-filipino

License: Hub field is other. These are research checkpoints trained on WikiText-TL-39 (Wikipedia-derived Tagalog text). This is not a legal clearance; check the source dataset and nanochat terms before reuse.

Repository layout

This repo is named for the selected depth D* = 20. Seed-0 confirmatory finals live in four folders only (no root-level weights):

Path	Depth	model_000294.pt SHA-256
d20/	20 (D*)	9e30fff3d6effc7c71af92e8488f9375a5d70cf1962ba371bee0e639836dde38
d16/	16	525301ebe3bc80875b31dd3f7fa19e12fc5405565f7b973811c0647168445cbf
d12/	12	5dfccc27b8b27c7c03faaeb92c1cbf1c884659b03795baa18921386d15e5277e
d8/	8	9c407f4fbc6f5bb2b40a36ae49fb38d088fa025b4f317e1ddaa629cc2068bbea

These are custom nanochat checkpoints (model_000294.pt + meta_000294.json), not transformers from_pretrained weights. Optimizer states and held-out test.jsonl are not included.

How to load: use nanochat at commit 92d63d4e8bb4df75c3b71618f31ddde2378b2bcd with the NANOCHAT_DATA_DIR hook from https://github.com/pageman/nanochat-filipino. Do not treat this as a chat model.