lrohAmca/nano-nla-qwen05b-20k-step800

Nano-NLA Qwen 0.5B, 20K RL Window, Step 800

This repository contains the first Nano-NLA checkpoint pair trained for Qwen/Qwen2.5-0.5B-Instruct.

It is a Natural Language Autoencoder checkpoint pair:

• av/: activation verbalizer / actor. This is a standard causal LM checkpoint.
• ar/: activation reconstructor / critic. This is a truncated Qwen backbone plus a bias-free vector head saved by Nano-NLA.

The model was trained on the first RL data window:

• Base model: Qwen/Qwen2.5-0.5B-Instruct
• Target layer: 16
• Residual width: 896
• RL row window: offset 0, max rows 20,000
• RL steps: 800
• Injection token id: 149705
• Injection scale: 25.0
• MSE scale: 29.93325909419153

Evaluation

Evaluation was run with scripts/run_eval.py --mode all on the final RL checkpoint. Summary results:

Metric	Value
Reconstruction FVE	0.90
Steganography, shuffle_bullets delta FVE	-0.0004
Steganography, coherence delta FVE	0.0000
Steganography, paragraph_summary delta FVE	-0.0006
Confabulation support rate	16.9%
Entity support rate	100%
Detail support rate	3.5%

These explanations are audit signals, not ground truth labels. At this scale the model can reconstruct activations well while still confabulating details in the natural-language explanation.

Loading

This is not a standalone Hugging Face text-generation model. It requires the Nano-NLA codebase because the AV uses activation-vector embedding injection and the AR uses the custom NLACriticModel wrapper.

Use it from the Nano-NLA repository after installing the project dependencies: IrohAmca/Nano-NLA

git clone https://github.com/IrohAmca/Nano-NLA.git
cd Nano-NLA
uv sync

Then download this checkpoint pair from Hugging Face:

from huggingface_hub import snapshot_download
from nano_nla.inference import NLAClient

repo = "lrohAmca/nano-nla-qwen05b-20k-step800"
local = snapshot_download(repo)

client = NLAClient(
    "configs/qwen05b.yaml",
    av_checkpoint=f"{local}/av",
    ar_checkpoint=f"{local}/ar",
    device="cuda:0",
)

If you already have a local Nano-NLA checkout, run the Python snippet from that checkout root so from nano_nla.inference import NLAClient resolves to the local package.

The nla_meta.yaml files under av/ and ar/ pin the prompt templates, injection token ids, injection_scale, and mse_scale used by this checkpoint.

Lineage

Nano-NLA is a compact Qwen 0.5B adaptation of Natural Language Autoencoders:

• Paper: https://transformer-circuits.pub/2026/nla/index.html
• Reference implementation: https://github.com/kitft/natural_language_autoencoders