README.md

---
library_name: sae_lens
tags:
- sae
- deepseek
- qwen
- interpretability
---

# Sparse Autoencoder (SAE) for deepseek-ai/DeepSeek-R1-Distill-Qwen-7B

This is a Sparse Autoencoder trained on layer 23 of `deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`.

> **⚠️ Important Note:** Due to a weight-folding bug in `sae_lens` when using Top-K with `normalize_activations: expected_average_only_in`, the `final_sae` weights in this run are corrupted. Please use the `best_sae` weights as shown in the usage example below, which were saved prior to the destructive folding step. When using `best_sae`, activations must be manually scaled by the `norm_scaling_factor` (0.09532437085610486) prior to encoding.

## Reproducibility & Configurations
- **Seed:** 42
- **Architecture:** topk
- **Expansion Factor:** 16
- **Top-K:** 64
- **Training Tokens:** 250000000
- **Learning Rate:** 0.0003

### Data
- **Mixtures:** Smoltalk + OpenThoughts
- **Total Prep Tokens:** 307322880
- **Fraction Reasoning:** 0.2679 achieved vs 0.5 configured
- **Fingerprint (SHA256):** `f474f0ff1226173b90aa88be3ae4dc8ea66a22f9e33b8e97709be70e959e50b8`

## Evaluation Metrics (Held-out Split)
- **L0:** 63.990631103515625
- **FVU (Fraction of Variance Unexplained):** 0.19430583889426564
- **Variance Explained (Reconstruction):** 0.8056941611057343
- **MSE (Per Token):** 662.951286315918
- **CE Loss (Clean):** 1.673828125
- **CE Loss (Spliced SAE):** 1.84197998046875
- **CE Loss (Zero Ablation):** 15.4091796875
- **CE Recovered Fraction:** 0.9877577319587629

## Validation / Usage

```python
from sae_lens import SAE
from transformer_lens import HookedTransformer
import torch

model = HookedTransformer.from_pretrained_no_processing("deepseek-ai/DeepSeek-R1-Distill-Qwen-7B", dtype=torch.bfloat16)
# NOTE: Load best_sae to avoid the weight-folding bug
sae = SAE.from_pretrained("<YOUR_HF_USERNAME>/sae-r1-distill-qwen7b-l23", "qwen7b-distill-l23-topk64-x16-smoltalk_seed42/best_sae")
```