analist/sae-r1-distill-qwen7b-l23

Sparse Autoencoder (SAE) for deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

This is a Sparse Autoencoder trained on layer 23 of deepseek-ai/DeepSeek-R1-Distill-Qwen-7B.

⚠️ Important Note: Due to a weight-folding bug in sae_lens when using Top-K with normalize_activations: expected_average_only_in, the final_sae weights in this run are corrupted. Please use the best_sae weights as shown in the usage example below, which were saved prior to the destructive folding step. When using best_sae, activations must be manually scaled by the norm_scaling_factor (0.09532437085610486) prior to encoding.

Reproducibility & Configurations

• Seed: 42
• Architecture: topk
• Expansion Factor: 16
• Top-K: 64
• Training Tokens: 250000000
• Learning Rate: 0.0003

Data

• Mixtures: Smoltalk + OpenThoughts
• Total Prep Tokens: 307322880
• Fraction Reasoning: 0.2679 achieved vs 0.5 configured
• Fingerprint (SHA256): f474f0ff1226173b90aa88be3ae4dc8ea66a22f9e33b8e97709be70e959e50b8

Evaluation Metrics (Held-out Split)

• L0: 63.990631103515625
• FVU (Fraction of Variance Unexplained): 0.19430583889426564
• Variance Explained (Reconstruction): 0.8056941611057343
• MSE (Per Token): 662.951286315918
• CE Loss (Clean): 1.673828125
• CE Loss (Spliced SAE): 1.84197998046875
• CE Loss (Zero Ablation): 15.4091796875
• CE Recovered Fraction: 0.9877577319587629

Validation / Usage

from sae_lens import SAE
from transformer_lens import HookedTransformer
import torch

model = HookedTransformer.from_pretrained_no_processing("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", dtype=torch.bfloat16)
# NOTE: Load best_sae to avoid the weight-folding bug
sae = SAE.from_pretrained("<YOUR_HF_USERNAME>/sae-r1-distill-qwen7b-l23", "qwen7b-distill-l23-topk64-x16-smoltalk_seed42/best_sae")