training-history/r002.md

# r002 — Pollock 1.0 training record

## Identity and provenance

| Field | Value |
|---|---|
| Revision / release | `r002` / Pollock 1.0 |
| Model ID | `SlayerLab/pollock-mini-lm-125m` |
| Published weights | [`5e571cde09ef2d1124e13d1c31d0b75f8bd0e513`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/5e571cde09ef2d1124e13d1c31d0b75f8bd0e513) |
| Source checkpoint | `runs/minimal-en-125m-4ep/ckpt.pt` |
| Checkpoint iteration / SHA-256 | 22,004 / `026f54a390036b35792aa8fb131c8b0d394efb6d933753fb0e1257e55a67374b` |
| `model.safetensors` SHA-256 | `2aada5b26abe9ce3b70af393d77be4ed1a799a7205b9c0dd5cfaff8bcaabce68` |
| nanoGPT commit | `3adf61e` |

Iteration 22,004 is one greater than configured `max_iters=22003` because of nanoGPT's inclusive loop/checkpoint behavior.

## Architecture and parameter accounting

| Setting | Value |
|---|---:|
| Type | GPT-2-style decoder-only Transformer |
| Layers / heads / width | 12 / 14 / 896 |
| Head dimension / MLP width | 64 / 3,584 |
| Context / vocabulary | 1,024 / 12,288 |
| Activation / dropout / bias | GELU / 0.0 / false |
| Position representation | learned absolute embeddings |
| Input/output token weights | tied |
| Transformer excluding embeddings | 115,627,904 |
| Tied token embedding/output matrix | 11,010,048 |
| nanoGPT reported count | 126,637,952 |
| Learned position embedding | 917,504 |
| Native unique trainable total | 127,555,456 |
| Transformers serialized total | 127,674,624 |
| Zero-valued compatibility bias parameters | 119,168 |

Conversion to `GPT2LMHeadModel` produced a maximum absolute logit difference of 0 on a deterministic 2×64-token probe.

## Tokenizer and dataset

The custom tokenizer is byte-level BPE with GPT-2-style pretokenization, 12,288 vocabulary entries, and 12,029 merges. `<|endoftext|>` (12285) is EOD/BOS/EOS/padding; `<|im_start|>` (12286) and `<|im_end|>` (12287) are additional special tokens. No chat template is defined. Tokenizer SHA-256: `6cda4e5ec8293f3b821e02253f9c0e88e43ad4f7c6e1324763e1fb91749fef51`.

Training used [Minimal EN Corpus 2.5B](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b), a subjectively selected mixture of 15 English sources with exact and approximate deduplication, language filtering, and benchmark decontamination.

| Split | Documents | BPE tokens | Binary SHA-256 |
|---|---:|---:|---|
| Train | 2,044,652 | 2,703,768,815 | `5396834c25664080c14793a4880f86b86ae94074a6036f8f54a58686b02d30dc` |
| Validation | 4,469 | 5,267,742 | `bf07bd1fb1f956a92ee7788f2a8110e1837f5b880ce432f6cd2a22d05f8ac135` |

## Complete training configuration

| Setting | Value |
|---|---:|
| Initialization | scratch |
| Final checkpoint iteration | 22,004 |
| Sequence / micro-batch per GPU | 1,024 / 12 |
| Global / per-GPU accumulation | 40 / 20 micro-steps |
| DDP world size / effective batch | 2 / 491,520 tokens |
| Token presentations / passes | 10,815,406,080 / approximately 4.0 |
| Optimizer / betas | fused AdamW / 0.9, 0.95 |
| Maximum / minimum LR | 3e-4 / 3e-5 |
| Schedule / warmup / decay | cosine / 440 / 22,003 iterations |
| Weight decay / grad clip | 0.1 / 1.0 |
| Evaluation interval / batches | 1,000 / 100 per split |
| Precision / compilation / backend | BF16 / `torch.compile` / NCCL |
| Seed | 1337 plus DDP-rank offset |
| Hardware | 2× NVIDIA GeForce RTX 4090 24 GB |
| PyTorch / runtime | 2.8.0+cu128 / approximately 12 h 41 min |

## Training-time validation

Loss estimates used 100 randomly sampled batches and are not a deterministic full-validation pass.

| Step | Train loss | Validation loss |
|---:|---:|---:|
| 0 | 9.5797 | 9.5732 |
| 1,000 | 3.4769 | 3.5219 |
| 2,000 | 3.0943 | 3.1266 |
| 5,000 | 2.8030 | 2.8569 |
| 10,000 | 2.6098 | 2.6928 |
| 15,000 | 2.5469 | 2.6056 |
| 18,000 | 2.5104 | 2.5790 |
| 19,000 | 2.4960 | 2.5795 |
| 20,000 | 2.4705 | **2.5600** |
| 21,000 | 2.4877 | 2.5653 |
| 22,000 | **2.4545** | 2.5775 |

The released weights are the final checkpoint. The increase after step 20,000 is within a noisy sampled protocol and is not definitive evidence of overfitting.

## English zero-shot benchmarks

All tasks used complete splits with `lm-evaluation-harness` 0.4.12, zero few-shot examples, batch size 8, BF16, 1,024 maximum context, and no sample limit. Random/NumPy/PyTorch/few-shot seeds were 0/1234/1234/1234. No requests were truncated.

| Benchmark | Primary | Additional | Samples |
|---|---:|---:|---:|
| BLiMP | acc 0.7669701493 | — | 67,000 |
| LAMBADA OpenAI | acc 0.2780904328 | perplexity 53.6675225106 | 5,153 |
| HellaSwag | acc_norm 0.2987452699 | acc 0.2818163712 | 10,042 |
| PIQA | acc_norm 0.6033732318 | acc 0.6137105550 | 1,838 |
| SciQ | acc_norm 0.6580000000 | acc 0.7370000000 | 1,000 |
| ARC-Easy | acc_norm 0.4229797980 | acc 0.4659090909 | 2,376 |
| ARC-Challenge | acc_norm 0.2414675768 | acc 0.2081911263 | 1,172 |

Structured results: [`../benchmarks/english.json`](../benchmarks/english.json). Release metadata: [`../release_manifest.json`](../release_manifest.json).

## Fixed inference reference

The shared [`fixed-sampling-v1`](../inference-samples/README.md) suite loads r002 from its immutable published commit, verifies the weight hash above, and records the exact output token IDs for the same four prompts used across r001-r006.

## Licensing and limitations

This is an English, unaligned base model, not an instruction-following assistant. It may hallucinate, generate harmful or biased content, and reproduce training patterns. It is unsuitable for high-risk or production decisions without independent safeguards. The dataset combines sources with different terms; see [`../LICENSE.md`](../LICENSE.md) and the dataset card.