# r003 - Pollock 1.1 training record

## Identity and provenance

| Field | Value |
|---|---|
| Revision / release | `r003` / Pollock 1.1 |
| Model ID | `SlayerLab/pollock-mini-lm-125m` |
| Published weights | [`698984b1d1b96c9b6ffaff00c7fc2e78e140e842`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/698984b1d1b96c9b6ffaff00c7fc2e78e140e842) |
| Source run | `runs/r003-lr4e-4-v4` |
| Source checkpoint | `checkpoints/ckpt.pt` |
| Checkpoint iteration / SHA-256 | 22,004 / `912483b44bb177c7a62fe19a53fc6db7b333cf881c57de15daf9dd7209d90a6b` |
| `model.safetensors` SHA-256 | `3a9964a0ba5f6d20ef6061d2a7e4442c1b001209ebe80f7b7148a25573decaa9` |
| Benchmark adapter SHA-256 | `0f53aa6b8454e856f8c4c489c7e6cbf207b8e124fe06bbb239315c017fe16fc0` |
| nanoGPT commit | `3adf61e154c3fe3fca428ad6bc3818b27a3b8291` (dirty workspace recorded) |
| Training environment | [DMPod GPT](https://github.com/dawidmajewski/dmpod-gpt) on RunPod |
| Run status | completed, exit code 0 |

The native checkpoint reports `iter_num=22004`. The wrapper's pre-run estimate in `config.json` used 22,003 target updates and is one effective batch lower because the legacy nanoGPT loop is inclusive. This record follows the native checkpoint and r002 release convention: 22,004 updates and 10,815,406,080 token presentations.

## Architecture and parameter accounting

| Setting | Value |
|---|---:|
| Type | GPT-2-style decoder-only Transformer |
| Layers / heads / width | 12 / 14 / 896 |
| Head dimension / MLP width | 64 / 3,584 |
| Context / vocabulary | 1,024 / 12,288 |
| Activation / dropout / bias | GELU / 0.0 / false |
| Position representation | learned absolute embeddings |
| Input/output token weights | tied |
| nanoGPT reported count | 126,637,952 |
| Native unique trainable total | 127,555,456 |

The architecture and parameter counts are unchanged from r002. The Transformers artifact has 127,674,624 unique serialized parameters, including 119,168 zero-valued compatibility bias parameters.

## Tokenizer and dataset

The custom tokenizer is byte-level BPE with GPT-2-style pretokenization, 12,288 vocabulary entries, and the special tokens `<|endoftext|>`, `<|im_start|>`, and `<|im_end|>`. Tokenizer SHA-256: `6cda4e5ec8293f3b821e02253f9c0e88e43ad4f7c6e1324763e1fb91749fef51`.

Training used [Minimal EN Corpus 2.5B](https://huggingface.co/datasets/SlayerLab/minimal-en-corpus-2.5b).

| Split | BPE tokens | Binary SHA-256 |
|---|---:|---|
| Train | 2,703,768,815 | `5396834c25664080c14793a4880f86b86ae94074a6036f8f54a58686b02d30dc` |
| Validation | 5,267,742 | `bf07bd1fb1f956a92ee7788f2a8110e1837f5b880ce432f6cd2a22d05f8ac135` |

Dataset and tokenizer identities match r002.

## Complete training configuration

| Setting | Value |
|---|---:|
| Initialization | scratch |
| Final checkpoint iteration | 22,004 |
| Sequence / micro-batch per GPU | 1,024 / 12 |
| Global / per-GPU accumulation | 40 / 20 micro-steps |
| DDP world size / effective batch | 2 / 491,520 tokens |
| Token presentations / passes | 10,815,406,080 / approximately 4.0 |
| Optimizer / betas | fused AdamW / 0.9, 0.95 |
| Maximum / minimum LR | 4e-4 / 4e-5 |
| Schedule / warmup / decay | cosine / 440 / 22,003 iterations |
| Weight decay / grad clip | 0.1 / 1.0 |
| Evaluation interval / batches | 1,000 / 100 per split |
| Evaluation subset | fixed 1,228,800 tokens per split |
| Training / data / evaluation seeds | 1,337 / 1,337 / 4,242 |
| Precision / compilation / backend | BF16 / `torch.compile` / NCCL |
| Hardware | 2x NVIDIA L40S |
| PyTorch / CUDA | 2.8.0+cu128 / 12.8 |
| Runtime | approximately 11 h 40 min |

The controlled change from r002 was the maximum/minimum learning rate. Hardware and the evaluation sampling implementation also differed operationally.

## Training-time validation

Loss estimates used fixed offset files containing 1,200 sequence starts per split. They are internally comparable within r003 but not directly comparable with r002's randomly sampled batches or r004's differently seeded offsets.

| Step | Train loss | Validation loss |
|---:|---:|---:|
| 0 | 9.5797 | 9.5732 |
| 1,000 | 3.3896 | 3.4346 |
| 2,000 | 3.0458 | 3.0779 |
| 3,000 | 2.8829 | 2.9362 |
| 4,000 | 2.8210 | 2.8694 |
| 5,000 | 2.7688 | 2.8231 |
| 6,000 | 2.6991 | 2.7486 |
| 7,000 | 2.6873 | 2.7129 |
| 8,000 | 2.6536 | 2.6828 |
| 9,000 | 2.6451 | 2.6794 |
| 10,000 | 2.5838 | 2.6674 |
| 11,000 | 2.5810 | 2.6154 |
| 12,000 | 2.5351 | 2.6060 |
| 13,000 | 2.5606 | 2.5888 |
| 14,000 | 2.5332 | 2.5935 |
| 15,000 | 2.5207 | 2.5801 |
| 16,000 | 2.4850 | 2.5726 |
| 17,000 | 2.4773 | 2.5945 |
| 18,000 | 2.4851 | 2.5548 |
| 19,000 | 2.4698 | 2.5561 |
| 20,000 | 2.4437 | **2.5362** |
| 21,000 | 2.4602 | 2.5417 |
| 22,000 | **2.4282** | 2.5529 |

The final recorded value is 2.5528755188 with perplexity 12.8439838496. The released source checkpoint is the final checkpoint, not the step-20,000 sampled minimum.

## English zero-shot benchmarks

Tasks used complete splits with `lm-evaluation-harness` 0.4.12, zero few-shot examples, batch size 8, BF16, and 1,024 maximum context. The adapter recorded 80 requests truncated to the model context.

| Benchmark | Primary | Additional | Samples | Delta vs r002 |
|---|---:|---:|---:|---:|
| BLiMP | acc 0.7773880597 | - | 67,000 | +0.010418 |
| LAMBADA OpenAI | acc 0.2899281972 | perplexity 47.7011402302 | 5,153 | +0.011838 acc |
| HellaSwag | acc_norm 0.3034256124 | acc 0.2850029875 | 10,042 | +0.004680 |
| PIQA | acc_norm 0.6109902067 | acc 0.6147986942 | 1,838 | +0.007617 |
| SciQ | acc_norm 0.6590000000 | acc 0.7660000000 | 1,000 | +0.001000 |
| ARC-Easy | acc_norm 0.4280303030 | acc 0.4861111111 | 2,376 | +0.005051 |
| ARC-Challenge | acc_norm 0.2534129693 | acc 0.2201365188 | 1,172 | +0.011945 |

All seven primary scores improved over r002. LAMBADA perplexity decreased by 5.966382. Because the r002 artifact reported zero truncated requests, the cross-revision comparison is informative but not a byte-for-byte identical evaluation protocol.

Structured English results: [`../benchmarks/english.json`](../benchmarks/english.json).

## Fixed inference reference

The shared [`fixed-sampling-v1`](../inference-samples/README.md) suite loads r003 from its immutable published commit, verifies the weight hash above, and records the exact output token IDs for the same four prompts used across r001-r006.

## Release preparation status

- Native training checkpoint verified against the backup archive.
- Benchmark adapter hash verified against the benchmark result record.
- Dataset, tokenizer, source snapshots, and fixed evaluation offsets are present.
- Transformers 5.15.1 conversion is complete; the `[2, 64]` deterministic parity probe produced a maximum absolute logit error of 0.0, and artifact hashes are recorded in `release_manifest.json`.
- Published and remotely verified at [`698984b1d1b96c9b6ffaff00c7fc2e78e140e842`](https://huggingface.co/SlayerLab/pollock-mini-lm-125m/commit/698984b1d1b96c9b6ffaff00c7fc2e78e140e842).

## Licensing and limitations

This is an English, unaligned base model, not an instruction-following assistant. It may hallucinate, generate harmful or biased content, and reproduce training patterns. It is unsuitable for high-risk or production decisions without independent safeguards. The dataset combines sources with different terms; see the dataset card and the model repository's `LICENSE.md`.