| --- |
| license: mit |
| pipeline_tag: text-generation |
| datasets: |
| - disco-jack-basement/byob-pd-book-corpus |
| language: |
| - en |
| - fr |
| - de |
| - it |
| - es |
| - pt |
| - nl |
| - sv |
| - da |
| - 'no' |
| - fi |
| - pl |
| - hu |
| - cs |
| - la |
| - el |
| - grc |
| - uk |
| tags: |
| - pytorch |
| - gpt |
| - transformer |
| - character-level |
| - language-model |
| - from-scratch |
| - public-domain |
| --- |
| |
| # letterpress - trained character-level GPTs |
|
|
| The trained checkpoints of [letterpress](https://github.com/Novotarskyi/letterpress), |
| a character-level GPT built from scratch in PyTorch (following Karpathy's "Let's |
| build GPT" lecture) and trained on nested tiers of a public-domain book corpus. |
| These are **base models**: they continue text in the style of the books they read, |
| one character at a time - they are not chatbots and do not follow instructions. |
|
|
| (The checkpoint files keep the model family's historical name `byob-lm` - the |
| letterpress repo was formerly `byob_llm` - so paths, benchmark reports, and the |
| companion dataset stay stable.) |
|
|
| ## Models |
|
|
| | tier | params | layers x width | context | vocab | best val loss | held-out bpc | wikitext bpc | file | |
| |---|--:|---|--:|--:|--:|--:|--:|---| |
| | `nano` | 0.82M | 4 x 128 | 128 | 65 | - | **2.68** | 4.09 | `nano/shakespeare-nano.pt` | |
| | `medium` | 25.4M | 8 x 512 | 256 | 97 | 1.2278 | **1.69** | 2.91 | `medium/byob-lm.best.pt` | |
| | `large` | 49.7M | 10 x 640 | 384 | 198 | 1.1111 | **1.57** | 2.30 | `large/byob-lm.best.pt` | |
| | `xlarge` | 99.9M | 14 x 768 | 512 | 199 | 0.9411 | **1.31** | 1.88 | `xlarge/byob-lm.best.pt` | |
| | `2xlarge` | 202M | 16 x 1024 | 512 | 204 | 0.8725 | **1.17** | 1.71 | `2xlarge/byob-lm.best.pt` | |
|
|
| Held-out bpc = bits-per-char on public-domain books absent from every training |
| tier; wikitext = out-of-domain. Both measured by the repo's bundled `lm_bench` |
| harness (seed 1337, byte-normalized); full scorecards live in the repo under |
| `lm_bench/benchmarks/`. Lower is better; the corpus tiers are nested, so every |
| gain down the table is scale, not data luck. |
|
|
| ## How to use |
|
|
| The checkpoints are self-describing (`{model, config, stoi, itos}`, optimizer |
| state stripped) but unpickle the letterpress `GPTConfig`, so you need the repo's |
| source next to them: |
|
|
| ```bash |
| git clone https://github.com/Novotarskyi/letterpress.git && cd letterpress |
| python3 -m venv .venv && .venv/bin/pip install -r requirements.txt |
| hf download disco-jack-basement/letterpress 2xlarge/byob-lm.best.pt --local-dir models |
| |
| .venv/bin/python -m inference.sample --ckpt models/2xlarge/byob-lm.best.pt --prompt "ROMEO:" |
| .venv/bin/python -m inference.interact --ckpt models/2xlarge/byob-lm.best.pt |
| cd lm_bench && ../.venv/bin/python -m lm_bench run --model byob:../models/2xlarge/byob-lm.best.pt --tasks core |
| ``` |
|
|
| ## Training provenance |
|
|
| - **nano** - Tiny Shakespeare (1.1M chars); the lecture demo, trained locally |
| - **medium** - medium tier (523M chars); Apple-Silicon MPS, ~1 h |
| - **large** - large tier (1.08B chars); Apple-Silicon MPS, ~3.5 h |
| - **xlarge** - xlarge tier (2.05B chars); rented H100 SXM, ~2.5 h |
| - **2xlarge** - 2xlarge tier (4.26B chars); rented H200, ~12.6 h |
|
|
| Each `<tier>/` folder here also carries the archive's provenance files |
| (context.md with the full hyperparameters and val curve, corpus_stats.txt, |
| corpus.lock.json, corpus_index.md) where the archive has them. |
|
|
| ## Data |
|
|
| Trained exclusively on the companion public-domain corpus |
| [`disco-jack-basement/byob-pd-book-corpus`](https://huggingface.co/datasets/disco-jack-basement/byob-pd-book-corpus) |
| (CC0; Project Gutenberg, Standard Ebooks, Internet Archive, Wikisource), curated |
| under a strict, code-enforced no-Russian-content rule. |
|
|
| ## License |
|
|
| Code and weights: MIT. The training corpus is CC0 and published separately. |
|
|
