shijtang/mini-nla-qwen2.5-0.5b

Mini NLA Qwen2.5-0.5B

This repository contains a small-scale reimplementation of a natural language autoencoder built around Qwen2.5-0.5B-Instruct. It maps a hidden-state activation to a natural-language explanation with an Activation Verbalizer (AV), then reconstructs the activation with an Activation Reconstructor (AR).

This is a custom PyTorch model bundle, not a standard text-generation checkpoint that can be loaded directly with AutoModelForCausalLM.

Contents

• base_model/: complete Qwen2.5-0.5B-Instruct weights, configuration, and tokenizer.
• av_sft/model.pt: supervised Activation Verbalizer checkpoint.
• ar_critic/model.pt: final Activation Reconstructor checkpoint.
• av_rl/model.pt: GRPO-optimized Activation Verbalizer checkpoint.
• data/: AV, AR, RL, and validation activation datasets.

The activation dimension is 896. Training used 1,024 AV examples, 1,024 AR examples, 2,048 RL examples, and 256 validation examples. Natural-language targets were generated with Qwen2.5-32B-Instruct.

Loading

The checkpoints are loaded by the mini_nla implementation:

from mini_nla.models_av import ActivationVerbalizer
from mini_nla.utils import load_state_dict_if_exists

av = ActivationVerbalizer("base_model", activation_dim=896, prefix_tokens=8)
load_state_dict_if_exists(av, "av_rl")

The AR is loaded similarly with CriticStyleActivationReconstructor.

Limitations

The learned roundtrip has negative held-out FVE and does not outperform mean-activation prediction. AR retains only a small fraction of sample-specific activation variance, and GRPO can exploit this weak reconstructor by converging toward generic explanation templates.