---
license: mit
language: [en]
tags: [url, compression, arithmetic-coding, nanogpt, int4]
datasets: [ks46/urls-tokenized]
---

# urls-250m — a URL language model for lossless compression

246,087,680 parameters: 12 layers × 1280 wide, 20 heads, SwiGLU MLP,
tied 8,192-entry byte-level BPE vocabulary (structural pre-split, hosts stored TLD-first),
learned absolute positions reset at every `<eos>`, so each URL is its own context.
Shipped as a 124.8 MiB int4-g64 artifact (`.nurl` v3: 34 bytes per 64 weights, f16 scales)
that the browser codec streams with exact integer dot products, making encode and decode bit-identical.

Objective: bits per character on held-out URLs (the compression ratio), decoded on the client in under a second.

## Training

| | |
|---|---|
| data | `ks46/urls-tokenized` shards 0–39, 19.9B tokens ≈ 561M URLs, one pass, seeded order |
| optimizer | AdamW β(0.9, 0.95), weight decay 0.01, peak LR 0.0006, warmup 1000, linear decay over the last 10% |
| batch | 64 windows × 512 tokens × 8 GPUs = 262,144 tokens per iteration, 76,000 iterations |
| tail | 7,600 quantisation-aware iterations on the exact int4-g64 grid at LR 6e-05 |
| hardware | 8× RTX 5090 (PCIe, no NVLink), bf16 autocast, ~680k tokens/s |
| code | linklet `training/` at git `unknown` |

## Checkpoints

`checkpoints/ckpt-i<iter>.pt` are immutable snapshots every 4,000 iterations (1.05B tokens), stable-phase
weights with optimizer state, loadable by `training/train.py` (`run.init=`) for decay or fine-tuning.
`ckpt-fp.pt` is the end of the full-precision phase; `ckpt.pt` is the final model after the QAT tail.
Eval is bits/char on a fixed 2,000-URL held-out slice (shard 512, even-stride sample).

| file | tokens | eval bits/char |
|---|---|---|
| ckpt-i004000.pt | 1.05B | nan |
| ckpt-i020000.pt | 5.24B | nan |
| ckpt-i040000.pt | 10.49B | nan |
| ckpt-i060000.pt | 15.73B | nan |
| ckpt-i068000.pt | 17.83B | nan |
| ckpt-i072000.pt | 18.87B | nan |
| ckpt-i080000.pt | 20.97B | nan |

## Final evaluation

```
fp32: 1.2629 bits/char over 99926 URLs (74 skipped)
kernel_sim(int4): 1.26372 bits/char
kernel: 1.2952 bits/char (rust/nurlcheck) over 500 URLs, artifact 124.8 MiB

```

`model/linklet-target-base.nurl` is the packed artifact; `model/tokenizer/` the tokenizer.
Every number above is copied from `manifest.json`, the run's provenance record.