Solshine/deception-v4-saes-qwen3-0p6b-base-v4mixed

V4 Sandbagging SAEs — Qwen3-0.6B-Base (V4 sandbagging mixed)

TopK (k=32) sparse autoencoders trained on residual-stream activations of Qwen/Qwen3-0.6B-Base under the V4 sandbagging-concealment prompt set (neutral, V0 threat, V3 threat).

Part of the ongoing research at deception-nanochat-sae-research.

Model specs

Property	Value
Base model	Qwen/Qwen3-0.6B-Base
Parameters	600M
Hidden dim (d_model)	1024
Total layers	28
Architecture	TopK SAE
Expansion factor	4 (d_sae = 4096)
Top-k	32
Dtype	float32

Available layers

Layer	Tag	MSE	Explained var	L0	Alive features
L14	qwen3_0p6b_base_topk_k32_L14_v4mixed	0.0531	1.000	32	457
L21	qwen3_0p6b_base_topk_k32_L21_v4mixed	0.1979	1.000	32	402

Research context

Qwen3 family (base — no instruction tuning). Does NOT sandbag to V0 threat (V0 acc 28% > neutral 22%), matching the Qwen2.5-1.5B pattern. V3 'Lie' sandbag works (V3 acc 19%). V0-direction steering produces NEGATIVE effect across all layers because direction is inverted. Publishing the SAEs for cross-model comparison; deferred-direction experiments from this family should use V3-sourced vectors.

These SAEs were trained as part of investigating whether directional steering and sparse-feature interventions can rescue trivia-answering capability under explicit sandbagging/threat prompts. See the parent repo's RESULTS_INDEX.md §63–§79 for the directional-steering analysis, and the papers/specificity_gap/manuscript.md for the full writeup.

Training data

• Prompts: 50 trivia questions × 3 templates (neutral, V0 threat, V3 'Lie about the answer')
• Activations: Residual-stream post-block output at all token positions (~3000–6000 vectors per layer)
• Epochs: 200 with Adam lr=3e-4, batch=128, decoder-row normalization per step

Loading

from huggingface_hub import hf_hub_download
import torch, json

repo_id = 'Solshine/deception-v4-saes-qwen3-0p6b-base-v4mixed'
tag = 'qwen3_0p6b_base_topk_k32_L14_v4mixed'  # example

state_path = hf_hub_download(repo_id, f'{tag}.pt')
cfg_path = hf_hub_download(repo_id, f'{tag}_cfg.json')
cfg = json.load(open(cfg_path))
state_dict = torch.load(state_path, weights_only=True)

The companion *_cfg.json files are SAELens-compatible — keys include model_name, hook_name, hook_layer, d_in, d_sae, architecture, k, normalize_activations.

Caveats

• Trained on a small activation set (~3–6k vectors per layer). These SAEs are optimized for the v4 sandbagging research pipeline and may not transfer cleanly to generic text without further training.
• MSE is evaluated on the training activations; held-out reconstruction quality was not separately measured.
• Not evaluated with Neuronpedia tooling — companion cfg is SAELens-standard but feature-level Cohen's-d statistics were not computed (unlike the earlier 393-SAE cohort).

License

Apache-2.0. See parent research repo for attribution.

Citation

If you use these SAEs, please cite:

@misc{deleeuw2026specificitygap,
  author = {DeLeeuw, Caleb},
  title = {Deception Research: Specificity Gap Between Circuit-Level and Feature-Level Interventions},
  year = {2026},
  url = {https://github.com/SolshineCode/deception-nanochat-sae-research}
}