---
license: mit
library_name: symbolic-ai
tags:
  - language-model
  - baseline
  - symbolic-ai
datasets:
  - HuggingFaceFW/fineweb-edu
---

# symbolic-ai-001-lm-baseline

The **conventional-pretraining control** for
[`jacob-valdez/symbolic-ai-001-3b46de3`](https://huggingface.co/jacob-valdez/symbolic-ai-001-3b46de3):
the *identical* 13.97M-parameter architecture, the *identical* 8,000-step budget
and optimizer, trained on the FineWeb-Edu corpus component alone.

It exists to answer one question: how much of an agent does ordinary next-token
pretraining give you, at fixed architecture and compute?

**Answer, on held-out seeds:** better language modelling (perplexity **51.9** vs
89.1) and an agent at or below the random policy on every interactive family —
gridworld 0.00 success, tool-calling 0.00 solved, arithmetic 0.00 exact,
MiniGrid 0.28 vs random 0.40.

## Provenance

| | |
|---|---|
| source repository | `symbolic-ai-001`, snapshot in [the companion repo](https://huggingface.co/jacob-valdez/symbolic-ai-001-3b46de3/tree/main/code) |
| source commit | `3b46de309a53f5afc2c5018a1116096de95fbf31` (`3b46de3`) |
| adapter version | 0.1.0 |
| corpus | `HuggingFaceFW/fineweb-edu`, `sample-10BT`, revision `87f09149ef4734204d70ed1d046ddc9ca3f2b8f9`, shard `sample/10BT/000_00000.parquet` |
| split rule | `sha256("symbolic-ai-001\|" + doc_id) < 0.01 → validation` |
| training | 8,000 steps × batch 24 × 512 = 98.3M tokens; AdamW, cosine, bf16; 22.2 min on one NVIDIA GB10 |
| held-out | val NLL 3.950, perplexity 51.9 |

Same tokenizer (`tokenizer.json`), same config shape (`baseline-lm-only.yaml`),
same evaluation protocol. Full analysis: `code/docs/symbolic-ai-001.md` in the
companion repo.

## License

MIT. Trained from random initialization.