Upload token_contract.md

token_contract.md

@@ -0,0 +1,197 @@
+# Token contract — `llama-3-audio-tokenizer`
+
+The authoritative description of what every token ID **means** in this project's
+compiled data, checkpoints and eval paths. Generated from the tokenizer itself,
+not transcribed by hand.
+
+    tokenizer:   checkpoints/llama-3-audio-tokenizer
+    vocab size:  156,960
+    fingerprint: b645cf6612315393eea21d29e94304bdbcbfee131a25de5eecf6bae6ee9f39c5
+    base model:  Llama-3.2-3B (128,000 BPE + 256 Llama specials)
+
+**A token ID is meaningless without this contract.** Compiled parquet stores raw
+integers; nothing in the data says which tokenizer minted them. Compiling with one
+tokenizer and training with another is silent — training runs, loss descends, and
+the model learns the wrong symbol for every audio frame. That is not hypothetical:
+it is exactly how the gemma3 SNAC base mismatch produced undecodable checkpoints,
+caught only at eval. Hence the fingerprint, stamped at compile time and asserted at
+training startup and in the eval decode path (`scripts/token_contract.py`).
+
+---
+
+## 1. ID map
+
+| range | count | what it is |
+|---|---:|---|
+| `0 – 127,999` | 128,000 | Llama-3 BPE text vocabulary |
+| `128,000 – 128,255` | 256 | stock Llama-3 specials (`<\|begin_of_text\|>`, `<\|eot_id\|>`, `<\|reserved_special_token_N\|>` …) |
+| `128,256 – 128,265` | 10 | **project control tokens** (§2) |
+| `128,266 – 156,937` | 28,672 | **SNAC audio codes** — 7 codebooks × 4,096 (§3) |
+| `156,938 – 156,959` | 22 | **conditioning + paralinguistic tokens** (§4) |
+
+28,960 added tokens in total (28,672 SNAC + 288 non-SNAC).
+
+Note the layout is *discontinuous*: the control block sits immediately below the
+audio band and the conditioning block immediately above it. Anything that assumes
+"all added tokens are contiguous above the base" is wrong.
+
+---
+
+## 2. Control tokens — `128,256 – 128,265`
+
+| id | token | role |
+|---:|---|---|
+| 128256 | `<\|reserved_0\|>` | unused |
+| **128257** | `<\|start_of_speech\|>` | opens the audio span; the model emits this itself as its first generated token |
+| **128258** | `<\|end_of_speech\|>` | closes the audio span |
+| **128259** | `<\|start_of_human\|>` | opens the prompt turn |
+| **128260** | `<\|end_of_human\|>` | closes the prompt turn |
+| **128261** | `<\|start_of_ai\|>` | opens the model turn |
+| **128262** | `<\|end_of_ai\|>` | closes the model turn |
+| 128263 | `<\|pad\|>` | padding |
+| 128264 | `<\|reserved_8\|>` | unused |
+| 128265 | `<\|reserved_9\|>` | unused |
+
+---
+
+## 3. Audio band — `128,266 – 156,937`
+
+    audio_token_base_id = 128266
+    codebooks           = 7
+    codebook size       = 4096
+    total audio tokens  = 28672        (base .. base + 7*4096 - 1 = 156937)
+
+SNAC 24 kHz emits three codebooks at a 1:2:4 temporal ratio:
+
+    c0: [seq_len]     12 Hz   coarsest
+    c1: [2*seq_len]   23 Hz
+    c2: [4*seq_len]   47 Hz   finest
+
+These are flattened to **7 tokens per frame**, in this fixed interleave:
+
+    frame i -> [ c0[i], c1[2i], c2[4i], c2[4i+1], c1[2i+1], c2[4i+2], c2[4i+3] ]
+
+Each raw code (0–4095) is offset by its **position in the frame**, not by which
+codebook it came from:
+
+| frame position | source | offset | id range |
+|---:|---|---|---|
+| 0 | `c0[i]` | base + 0×4096 | 128,266 – 132,361 |
+| 1 | `c1[2i]` | base + 1×4096 | 132,362 – 136,457 |
+| 2 | `c2[4i]` | base + 2×4096 | 136,458 – 140,553 |
+| 3 | `c2[4i+1]` | base + 3×4096 | 140,554 – 144,649 |
+| 4 | `c1[2i+1]` | base + 4×4096 | 144,650 – 148,745 |
+| 5 | `c2[4i+2]` | base + 5×4096 | 148,746 – 152,841 |
+| 6 | `c2[4i+3]` | base + 6×4096 | 152,842 – 156,937 |
+
+So positions 1 and 4 are both `c1`, and 2/3/5/6 are all `c2` — the offset encodes
+*where in the frame* a code sits, which is what makes the stream decodable without
+a separate structure signal.
+
+**Consecutive duplicate frames are removed** at encode time (frames sharing the
+same `c0`), so token count is not exactly proportional to duration.
+
+Derived constants (`scripts/snac_tokenizer.py`):
+
+    samples per c0 frame  = 512      (encoder_rates [2,4,8,8] x vq_stride 4)
+    streaming window      = 4 frames = 28 tokens, middle frame kept
+    long-audio window     = 512 frames (~43.7 s), 4-frame context each side
+
+Rule of thumb used throughout the project: **82 tokens ≈ 1 second of audio.**
+
+---
+
+## 4. Conditioning + paralinguistics — `156,938 – 156,959`
+
+| id | token | role |
+|---:|---|---|
+| 156938 / 156939 | `<\|speaker>` / `<speaker\|>` | wrap a speaker name |
+| 156940 / 156941 | `<\|style>` / `<style\|>` | wrap a style label (rasa) or accent string (globe) |
+| 156942 / 156943 | `<\|env>` / `<env\|>` | wrap an environment label; open class, label stays free text |
+
+Non-verbals — a **closed set of 16**, one token each (no wrapper):
+
+| id | token | | id | token |
+|---:|---|---|---:|---|
+| 156944 | `<\|nv_breath\|>` | | 156952 | `<\|nv_hum\|>` |
+| 156945 | `<\|nv_stammer\|>` | | 156953 | `<\|nv_gasp\|>` |
+| 156946 | `<\|nv_throat\|>` | | 156954 | `<\|nv_wheeze\|>` |
+| 156947 | `<\|nv_laugh\|>` | | 156955 | `<\|nv_sneeze\|>` |
+| 156948 | `<\|nv_swallow\|>` | | 156956 | `<\|nv_snort\|>` |
+| 156949 | `<\|nv_sniff\|>` | | 156957 | `<\|nv_yawn\|>` |
+| 156950 | `<\|nv_sigh\|>` | | 156958 | `<\|nv_groan\|>` |
+| 156951 | `<\|nv_cough\|>` | | 156959 | `<\|nv_burp\|>` |
+
+The asymmetry is deliberate: non-verbals are a closed vocabulary and get dedicated
+tokens; environment labels are an open class, so only the wrapper is a token and
+the label inside stays free text — a new environment class needs no tokenizer
+change. Source-form mapping (`<breath>`, `[bird_squawk]`, …) is in
+`scripts/paralinguistics.py`.
+
+---
+
+## 5. Sequence layouts
+
+Built by `scripts/chat_templates.py`. `prompt_end` is defined as everything up to
+and **including** `<|start_of_ai|>` — the prompt therefore stops *before*
+`<|start_of_speech|>`, which the model emits itself.
+
+**TTS with conditioning** (rasa, globe, bhili):
+
+    <|start_of_human|><|begin_of_text|>
+      <|speaker>NAME<speaker|>\n
+      <|style>LABEL<style|>\n
+      TEXT
+    <|eot_id|><|end_of_human|><|start_of_ai|>
+      <|start_of_speech|> ...audio... <|end_of_speech|>
+    <|end_of_ai|>
+
+Metadata order is always **speaker → style → accent**, each block followed by a
+newline. An empty value emits nothing at all (no empty wrapper).
+
+**Multi-turn conversation** (`gemini_vc_conversational`) — no metadata prefix;
+speaker labels are inline turn markers inside the text:
+
+    <|start_of_human|><|begin_of_text|>
+      <|speaker>A<speaker|>\nturn one\n\n<|speaker>B<speaker|>\nturn two ...
+    <|eot_id|><|end_of_human|><|start_of_ai|><|start_of_speech|> ...
+
+Turn separator is a **blank line** (`\n\n`) before each subsequent
+`<|speaker>` marker. `normalize_text` preserves newline runs (the old
+collapse-to-one-space behavior was removed when the `\n\n` issue was fixed),
+so the separator reaches the model verbatim; verified present in 100% of
+compiled multi-turn rows in both `gemini_vc` and `gemini_src_conv`. The `\n`
+after `<speaker|>` also survives. Inference prompts must match this form.
+
+---
+
+## 6. Fingerprint
+
+`scripts/token_contract.py::compute_fingerprint` is a sha256 over exactly the
+things that change what an ID means:
+
+- the full added-token map (content → id), sorted
+- vocab size
+- the SNAC base id and layout constants (codebooks × codebook size)
+
+It deliberately **excludes** `tokenizer_config.json` niceties — padding side, chat
+template, `model_max_length` — because those do not change a token's meaning and
+including them would fire on cosmetic edits.
+
+    current fingerprint: b645cf6612315393eea21d29e94304bdbcbfee131a25de5eecf6bae6ee9f39c5
+
+Why a content hash and not a range check: the audio-band range check in
+`snac_tokenizer.decode_audio` catches the loud failure (IDs outside the band). It
+cannot catch the quiet one — sibling tokenizers `llama-3-audio-tokenizer`
+(156,938), `-tok_trimmed` (156,942) and `-style` (156,952) all share base 128,266,
+so every range check passes while `<|style>` means something different in the data
+than in the model.
+
+Mismatch is a hard error, never a warning. A silent wrong-tokenizer run costs a
+full training cycle.
+
+---
+
+*Generated from `checkpoints/llama-3-audio-tokenizer` with `scripts/token_contract.py`;
+layout constants from `scripts/snac_tokenizer.py`, sequence templates from
+`scripts/chat_templates.py`.*