File size: 1,772 Bytes
d5bc67a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 | ---
license: mit
library_name: symbolic-ai
tags:
- language-model
- baseline
- symbolic-ai
datasets:
- HuggingFaceFW/fineweb-edu
---
# symbolic-ai-001-lm-baseline
The **conventional-pretraining control** for
[`jacob-valdez/symbolic-ai-001-3b46de3`](https://huggingface.co/jacob-valdez/symbolic-ai-001-3b46de3):
the *identical* 13.97M-parameter architecture, the *identical* 8,000-step budget
and optimizer, trained on the FineWeb-Edu corpus component alone.
It exists to answer one question: how much of an agent does ordinary next-token
pretraining give you, at fixed architecture and compute?
**Answer, on held-out seeds:** better language modelling (perplexity **51.9** vs
89.1) and an agent at or below the random policy on every interactive family —
gridworld 0.00 success, tool-calling 0.00 solved, arithmetic 0.00 exact,
MiniGrid 0.28 vs random 0.40.
## Provenance
| | |
|---|---|
| source repository | `symbolic-ai-001`, snapshot in [the companion repo](https://huggingface.co/jacob-valdez/symbolic-ai-001-3b46de3/tree/main/code) |
| source commit | `3b46de309a53f5afc2c5018a1116096de95fbf31` (`3b46de3`) |
| adapter version | 0.1.0 |
| corpus | `HuggingFaceFW/fineweb-edu`, `sample-10BT`, revision `87f09149ef4734204d70ed1d046ddc9ca3f2b8f9`, shard `sample/10BT/000_00000.parquet` |
| split rule | `sha256("symbolic-ai-001\|" + doc_id) < 0.01 → validation` |
| training | 8,000 steps × batch 24 × 512 = 98.3M tokens; AdamW, cosine, bf16; 22.2 min on one NVIDIA GB10 |
| held-out | val NLL 3.950, perplexity 51.9 |
Same tokenizer (`tokenizer.json`), same config shape (`baseline-lm-only.yaml`),
same evaluation protocol. Full analysis: `code/docs/symbolic-ai-001.md` in the
companion repo.
## License
MIT. Trained from random initialization.
|