File size: 2,503 Bytes
aaa4b7f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | ---
license: mit
base_model: syvb/nanonla-qwen3-8b-L24-av
tags:
- nla
- natural-language-autoencoder
- qwen3
---
# nanonla-qwen3-8b-L24-av-multi16-singletag-bs64
**16-slot affine experiment arm.**
Continued AV-SFT warm-started from [`syvb/nanonla-qwen3-8b-L24-av`](https://huggingface.co/syvb/nanonla-qwen3-8b-L24-av)
(Qwen3-8B, injection layer 24, d_model 4096).
## Multi-input affine experiment
Hypothesis: repeating the injection marker **N times**, each slot getting its own
learned affine `A_i·v_norm + b_i` over the (normalized) injected activation, gives
the backbone N independently-readable "views" of one activation and improves
verbalization vs. the status-quo single injection.
- **Experiment arm:** N=16 markers, 16 full d×d+bias affines, identity-init,
full continued AV-SFT (affines **and** backbone trained).
- **Control arm:** N=1, no affine — same warm-start, same data, same step budget
(matched compute).
Both arms: 1000 steps, eff. batch 64, 1 epoch (64k rows), lr 2.5e-5, single-tag, `injection_scale = sqrt(d_model)`. Trained with the
standalone (miles-free) trainer `tools/train_av_standalone.py`.
## Held-out val NLL (lower is better)
Evaluated on **4001 doc-disjoint** held-out rows (val rows whose `doc_id`
does **not** appear in the training split — a true held-out set, not just a row
split), gold activation injected. Both arms scored on the **same** rows (paired).
| arm | val NLL/token | perplexity |
|---|---|---|
| **16-slot affine (experiment)** | 1.4580 | 4.298 |
| **1-slot (control)** | 1.4753 | 4.372 |
| **Δ (control − experiment)** | **+0.0172** | — |
Paired row-level bootstrap (10k resamples): Δ = **+0.0172** nats/token,
95% CI **[+0.0167, +0.0178]**.
✅ **The 16-slot affine significantly improves val NLL** (Δ=+0.0172, 95% CI [+0.0167, +0.0178] excludes 0).
### Caveats
- This is a **system-level** comparison (16 markers + per-slot affine vs. 1 marker).
It does **not** isolate the affine from the effect of simply repeating the
injection 16× — a "16 markers, no affine" arm would be needed for that.
- Continued warm-start, not a converged run (1000 steps, eff. batch 64, 1 epoch (64k rows), lr 2.5e-5, single-tag).
For the 16-slot model the per-slot affines are in `nla_affine.safetensors`
(`weight [16,4096,4096]`, `bias [16,4096]`); apply them at injection time — see
`launch/eval_av_val_loss.py --multi-input-slots 16 --affine-path ...`.
wandb: https://wandb.ai/octahedral-systems/nla-multi-affine-experiment
|