---
license: mit
base_model: syvb/nanonla-qwen3-8b-L24-av
tags:
- nla
- natural-language-autoencoder
- qwen3
---

# nanonla-qwen3-8b-L24-av-multi16-singletag-bs64

**16-slot affine experiment arm.**

Continued AV-SFT warm-started from [`syvb/nanonla-qwen3-8b-L24-av`](https://huggingface.co/syvb/nanonla-qwen3-8b-L24-av)
(Qwen3-8B, injection layer 24, d_model 4096).

## Multi-input affine experiment

Hypothesis: repeating the injection marker **N times**, each slot getting its own
learned affine `A_i·v_norm + b_i` over the (normalized) injected activation, gives
the backbone N independently-readable "views" of one activation and improves
verbalization vs. the status-quo single injection.

- **Experiment arm:** N=16 markers, 16 full d×d+bias affines, identity-init,
  full continued AV-SFT (affines **and** backbone trained).
- **Control arm:** N=1, no affine — same warm-start, same data, same step budget
  (matched compute).

Both arms: 1000 steps, eff. batch 64, 1 epoch (64k rows), lr 2.5e-5, single-tag, `injection_scale = sqrt(d_model)`. Trained with the
standalone (miles-free) trainer `tools/train_av_standalone.py`.

## Held-out val NLL (lower is better)

Evaluated on **4001 doc-disjoint** held-out rows (val rows whose `doc_id`
does **not** appear in the training split — a true held-out set, not just a row
split), gold activation injected. Both arms scored on the **same** rows (paired).

| arm | val NLL/token | perplexity |
|---|---|---|
| **16-slot affine (experiment)** | 1.4580 | 4.298 |
| **1-slot (control)** | 1.4753 | 4.372 |
| **Δ (control − experiment)** | **+0.0172** | — |

Paired row-level bootstrap (10k resamples): Δ = **+0.0172** nats/token,
95% CI **[+0.0167, +0.0178]**.

✅ **The 16-slot affine significantly improves val NLL** (Δ=+0.0172, 95% CI [+0.0167, +0.0178] excludes 0).

### Caveats
- This is a **system-level** comparison (16 markers + per-slot affine vs. 1 marker).
  It does **not** isolate the affine from the effect of simply repeating the
  injection 16× — a "16 markers, no affine" arm would be needed for that.
- Continued warm-start, not a converged run (1000 steps, eff. batch 64, 1 epoch (64k rows), lr 2.5e-5, single-tag).

For the 16-slot model the per-slot affines are in `nla_affine.safetensors`
(`weight [16,4096,4096]`, `bias [16,4096]`); apply them at injection time — see
`launch/eval_av_val_loss.py --multi-input-slots 16 --affine-path ...`.

wandb: https://wandb.ai/octahedral-systems/nla-multi-affine-experiment