---
base_model: Qwen/Qwen2.5-7B-Instruct
datasets:
- MaxChess/nla-affect-10k
library_name: transformers
license: apache-2.0
tags:
- natural-language-autoencoder
- activation-reconstructor
- interpretability
- affect
---

# Qwen2.5-7B NLA Affect AR

Experimental activation reconstructor (AR): explanation text to a 3,584-dimensional
Qwen2.5-7B-Instruct layer-20 activation direction. The backbone is truncated to 21
transformer blocks and uses a separately saved `value_head.safetensors`.

## Evaluation on 200 AR-held-out teacher explanations

- Directional FVE: 0.34277.
- Bootstrap 95% CI: [0.30129, 0.37560].
- Mean cosine similarity: 0.76460.
- Shuffled FVE: -0.77740.

Targets and predictions are independently L2-normalized. FVE is
`1 - mean(||h-h_hat||^2) / mean(||h-mean(h)||^2)` on normalized directions.

## Training

- 5,000 examples, one epoch, 209 optimizer steps.
- Direction-only normalized MSE, scale `sqrt(3584)`.
- Effective global batch: 24 on three V100-32GB GPUs.
- PyTorch SDPA.

This AR score measures reconstruction from teacher explanations. The current full
AV -> AR cycle is substantially weaker (FVE 0.03765), indicating that AV is the main
bottleneck. See `MaxChess/nla-affect-10k` for exact splits and source code.