Add model card

README.md

@@ -0,0 +1,173 @@
+---
+base_model: nvidia/personaplex-7b-v1
+library_name: peft
+license: other
+license_name: nvidia-open-model-license
+license_link: https://huggingface.co/nvidia/personaplex-7b-v1
+datasets:
+- JSALT2026-Conv-AI-Simulator/fisher-v1
+tags:
+- lora
+- personaplex
+- moshi
+- full-duplex
+- speech-to-speech
+- backchannel
+---
+
+# personaplex-fisher-lora-backchannel-v14b
+
+### Augmentation recipe
+
+| lever | value | effect |
+|---|---|---|
+| `--backchannel_aug_prob` | 0.3 | 30% of examples are augmented |
+| `--backchannel_aug_delete_share` | 1.0 | every augmented example is a *deletion* example — no insertions |
+| `--backchannel_aug_delete_prob` | 0.5 | half the backchannels in such an example are removed |
+| `--backchannel_aug_purge_prob` | 0.05 | 5% of examples have **every** backchannel erased, drawn independently of `aug_prob` |
+
+The purge is the important one. At `delete_prob 0.5` alone, every augmented example still contains
+backchannels, so the model can learn "fewer here" but never "none" — and "none" is the only signal
+that acts on *spontaneous* production. Purged examples keep their head positives, so head
+supervision is identical across all four v14 arms and the purge acts only on the LM.
+
+## Training
+
+* **Data:** [`JSALT2026-Conv-AI-Simulator/fisher-v1`](https://huggingface.co/datasets/JSALT2026-Conv-AI-Simulator/fisher-v1)
+  (Fisher telephone conversations), `max_length` 2000 frames (~160 s), 4 epochs, batch 1 x
+  grad-accum 16.
+* **Backchannel labels** are derived from word alignments (Fisher's own
+  `utterances[].backchannels` column is empty): each speaker is split into inter-pausal units, and
+  one is marked a backchannel when it is short (`max_duration 1.5`), passes a content gate, and did
+  not take the floor (`floor_gap 2.0`, retaining floor status across gaps up to 2 s).
+  `--no-backchannel_include_laughter` — unlike `v2`, listener laughter does **not** count.
+  `--backchannel_turn_clearance 1.5`, `--drop_non_speech_words`.
+* Only frames where a backchannel is *possible* (agent silent, partner talking,
+  `agent_silence_margin 0.24`) are supervised; the rest are ignored rather than taught as negatives.
+
+### Architecture
+
+* **LoRA:** attention only, on the temporal transformer —
+  `^decoder\.model\.layers\.\d+\.(self_attn\.(q_proj|k_proj|v_proj|o_proj)\.linear)$` —
+  `r=16`, `alpha=32`, `dropout=0.05`.
+* **Backchannel head:** `Linear(hidden -> 256) -> GELU -> Linear(256 -> 1)`, reading a single frame
+  of the trunk's hidden state. `context=0`, i.e. **no** temporal convolution — that is the `claims`
+  line of checkpoints.
+* **Trainable tokens:** the `<bc>` marker is token id **4**, the `<0x00>` byte-fallback slot — the
+  one slot the tokenizer can never emit (it occurs zero times across all 11,396 Fisher rows). Its
+  embedding row is trained in both `decoder.model.embed_tokens` and
+  `depth_decoder.text_embed_tokens`; the rest of both tables stays frozen.
+* **Loss:** focal (`gamma=2.0`, `alpha=0.9`), backchannel loss weight `1.0`, head bias initialised
+  to prior `0.016`, head LR `1e-3`, LM LR `2e-5`. Semantic codebook weight `1.0`, acoustic `0.02`,
+  pad-text loss weight `0.3`.
+
+## Evaluation
+
+Three measurements that answer three different questions:
+
+| measurement | what it asks | material |
+|---|---|---|
+| Fisher head eval | can the head rank frames? | Fisher validation, teacher-forced |
+| TurnBench head eval | does that survive out of domain, against human annotations? | TurnBench dev, teacher-forced |
+| self-play obedience | when the head fires, does the model produce a backchannel? | Fisher-domain self-play |
+
+Matching tolerance is 5 frames (400 ms) in both head evals — exact-frame agreement is too strict a
+bar for a phenomenon whose natural placement window is a few hundred ms wide.
+
+### Stage 1 — Fisher validation, teacher-forced
+
+93,042 supervised frames, 1,213 positive (1.30%).
+
+| Average Precision | ROC-AUC | mean p on positives | mean p on negatives |
+|---|---|---|---|
+| 0.352 | 0.928 | 0.486 | 0.182 |
+
+| Threshold | Precision | Recall | F1@400ms | Rate vs. human |
+|---|---|---|---|---|
+| 0.70 | 0.206 | 0.305 | 0.246 | 1.48x |
+
+
+### Stage 2 — TurnBench dev, out of domain
+
+12 conversations / 24 dialogue passes, 6,553 supervised frames, 97 positive, 35 reference events,
+human rate 1.52 events/min. Run on a **-23 LUFS loudness-normalised** copy: TurnBench ships 19.4 dB
+apart internally and ~15 dB below Fisher, which moves where the head fires.
+
+Two conditions: `keep` leaves the recording untouched (the agent's own backchannels stay in its
+context, so the head can lean on them); `remove` erases them, which is the condition that matches
+self-play and the render harness.
+
+| Condition | AP | ROC-AUC | Best lag | Rate-matched thr | P | R | **F1** | Rate |
+|---|---|---|---|---|---|---|---|---|
+| `keep` | 0.147 | 0.810 | -1 | 0.650 | 0.222 | 0.229 | **0.225** | 1.03x |
+| `remove` | 0.123 | 0.805 | -4 | 0.650 | 0.485 | 0.457 | **0.471** | 0.94x |
+
+At best-F1 rather than rate-matched, `remove` reaches **F1 0.475** (thr 0.675, P 0.583 / R 0.400) but
+sits at 0.69x the human rate — it buys precision by firing less.
+
+Two caveats worth carrying:
+
+* **The in-domain to out-of-domain drop is large and not checkpoint-specific**: AP 0.35 -> 0.12-0.15,
+  ROC-AUC 0.93 -> 0.81.
+* **v14b's optimal lag on `remove` is -4 frames** (320 ms *before* the human onset), where the other
+  arms sit at -1/-2. AP at that lag is 0.231, against 0.123 at lag 0.
+
+### Stage 3 — self-play obedience
+
+100 dialogues x 80 s windows (133 min) in the partner-backchannelling condition. Thresholds are
+calibrated **on generated streams**, never reused from validation — confidence is systematically
+different in free generation than under teacher forcing.
+
+| Condition | Threshold | Fires/min | **Obeyed** | **Realised** | clean bc | overrun | other | stranded |
+|---|---|---|---|---|---|---|---|---|
+| off | — | 0.00 | — | — | — | — | — | — |
+| low | 0.740 | 1.03 | **100%** | **77%** | 106 | 31 | 0 | 0 |
+| normal | 0.695 | 1.48 | **98%** | **69%** | 137 | 58 | 3 | 0 |
+| high | 0.615 | 3.59 | **95%** | **65%** | 311 | 142 | 24 | 2 |
+| max | 0.485 | 8.31 | **89%** | **65%** | 717 | 270 | 84 | 37 |
+| *Partner (within-dialogue baseline)* | — | *1.51* | — | — | — | — | — | — |
+
+* **`obeyed`** = the model started *something* at the trigger. **`realised`** = what it said was a
+  backchannel and stayed one. **`overrun`** = it began with a backchannel and then kept talking into
+  a turn. **`other`** = not a backchannel at all.
+* At `normal`, only **3 of 198** triggers produced a non-backchannel.
+* Obedience degrades gracefully: still 89% obeyed when driven to 5.5x the human rate.
+* **Realised forms** are the expected inventory: `yeah` (~500), `right`, `okay`, `uh-huh`, `mm`,
+  `mhm`.
+
+#### Controllability of `off`
+
+With the head off, **zero** forced events fire, but the model still emits **0.70 spontaneous
+lexical backchannels/min**, against 1.51/min for its partner in the same calls. So `off` means "no
+head-driven backchannels", not silence — the purge lever moves this number but does not zero it.
+
+#### Overrun
+
+29% of obeyed triggers at `normal` (58/198) overran into a turn. The forced-decode harness measures
+the same phenomenon far lower — 22/177 = 12% continued past a complete backchannel, of which 9
+(5.1%) were true turn attempts — because its gate requires 3.2 s of clearance before firing, so it
+only fires where a turn *cannot* start. Self-play lacks that constraint, so **treat 29% as the
+pessimistic bound and 5% as what a clearance-gated deployment sees.**
+
+### Choosing among the v14 arms
+
+Rate-matched F1 on TurnBench, and obedience at a matched ~1.5 fires/min:
+
+| ckpt | augmentation | Fisher F1 | TB `keep` F1 | TB `remove` F1 | obeyed | realised |
+|---|---|---|---|---|---|---|
+| v14a | none | .252 | **.267** | .348 | 87% | 51% |
+| **v14b** | **deletions only** | .246 | .225 | .471 | **98%** | **69%** |
+| v14c | insertions only | .244 | **.270** | .324 | 93% | 63% |
+| v14d | both | .246 | .257 | **.479** | 93% | 53% |
+
+1. **Fisher separates nothing** — all four within 0.008 F1 and 0.002 AUC on 1,213 positive frames.
+2. **`remove` splits them into two pairs**: v14b/v14d at .471/.479 against v14a/v14c at .348/.324, a
+   ~45% relative gap driven by precision. But v14b vs v14d is a coin flip on 35 reference events.
+3. **`keep` ranks them differently** and puts v14b last. `keep` is the easier condition and the one
+   that does not match how the model is actually run.
+4. **Obedience is where they separate, and v14b wins outright** — at every floor, on both metrics,
+   with 3 non-backchannels out of 198 against 26 for v14a.
+
+Counterintuitive and worth restating: **deletions-only beats insertions-only and both**, even though
+insertion is the arm meant to teach the model to obey a backchannel request.
+