| # Qwen 2.5 7B — Weight Forensics Report |
|
|
| > Comparison: `Qwen/Qwen2.5-7B-Instruct` (base) vs `qwen-2.5-7b-apostate` (Apostate) |
| > Method: Safetensors-level tensor comparison via `src/weight/` pipeline |
| > Date: 2026-06-02 |
|
|
| --- |
|
|
| ## Executive Summary |
|
|
| The Apostate abliteration of Qwen 2.5 7B is a **textbook weight-space orthogonal projection** — the classic abliteration method. **55 of 339 tensors** (16.2%) were modified, spanning 2.72B of the model's 7.62B parameters (35.8% of total parameter count). The edits follow the standard refusal-direction-removal pattern: |
|
|
| 1. **`mlp.down_proj.weight`** — 27 of 28 layers (all except layer 11) |
| 2. **`self_attn.o_proj.weight`** — 27 of 28 layers (all except layer 11) |
| 3. **`model.embed_tokens.weight`** — 1 tensor (minimal edit) |
|
|
| Layer 11 was entirely untouched. The edit magnitude follows a distinctive three-phase pattern: low in early layers (0–5), elevated in middle layers (6–20), and reduced again in late layers (21–27). This is the signature of Apostate's `balanced` profile targeting the model's "refusal circuits" concentrated in mid-to-late-middle layers. |
|
|
| --- |
|
|
| ## Model Architecture |
|
|
| | Property | Value | |
| |---|---| |
| | Architecture | `Qwen2ForCausalLM` | |
| | Parameters | 7.62B | |
| | Layers | 28 | |
| | Hidden size | 3584 | |
| | Attention heads | 28 (GQA: 4 KV heads) | |
| | Intermediate size | 18944 | |
| | Vocabulary | 152,064 | |
| | Tie word embeddings | false (both models) | |
| | Thinking model | **No** — standard causal LM | |
| | File size (base) | ~15 GB (4-shard safetensors) | |
| | File size (apostate) | ~15 GB (single safetensors) | |
|
|
| --- |
|
|
| ## Tensor Comparison |
|
|
| ### Overview |
|
|
| | Metric | Value | |
| |---|---| |
| | Total tensors | 339 | |
| | Changed tensors | **55 (16.2%)** | |
| | Unchanged tensors | 284 (83.8%) | |
| | Total parameters | 7,615,616,512 | |
| | Parameters changed | **2,724,986,880 (35.8%)** | |
| | Layers modified | 27 of 28 (layer 11 skipped) | |
|
|
| ### Changed Tensor Breakdown |
|
|
| | Tensor Type | Count | Edit Norm (mean) | Rel Norm (mean) | Layers | |
| |---|---|---|---|---| |
| | `mlp.down_proj.weight` | 27 | 2.270 | 0.017 | 0–10, 12–27 | |
| | `self_attn.o_proj.weight` | 27 | 1.036 | 0.018 | 0–10, 12–27 | |
| | `model.embed_tokens.weight` | 1 | 0.278 | 0.001 | — | |
| | **Total** | **55** | — | — | — | |
|
|
| ### Edit Magnitude Statistics |
|
|
| | Statistic | Value | |
| |---|---| |
| | Edit norm mean | 1.628 | |
| | Edit norm median | 1.364 | |
| | Edit norm P25 | 0.810 | |
| | Edit norm P75 | 2.448 | |
| | Edit norm P95 | 3.455 | |
| | Relative edit mean | 1.72% | |
| | Relative edit median | 2.00% | |
|
|
| --- |
|
|
| ## Per-Layer Analysis |
|
|
| ### Layer-Level Edit Profile |
|
|
| The 27 modified layers show a distinctive **three-phase pattern** in edit magnitude: |
|
|
| ``` |
| Layer Edit Norm (down_proj / o_proj) Phase |
| ───────────────────────────────────────────────── |
| 0 1.27 / 0.49 ▓▓ Early (low) |
| 1 1.12 / 0.60 ▓▓ |
| 2 1.28 / 0.59 ▓▓ |
| 3 1.48 / 0.61 ▓▓ |
| 4 1.42 / 0.63 ▓▓ |
| 5 1.48 / 0.61 ▓▓ |
| 6 2.55 / 1.22 ▓▓▓▓ Mid (high) |
| 7 2.68 / 1.29 ▓▓▓▓ |
| 8 2.70 / 1.16 ▓▓▓▓ |
| 9 2.45 / 1.27 ▓▓▓▓ |
| 10 2.69 / 1.23 ▓▓▓▓ |
| 11 — SKIPPED — ·· |
| 12 2.89 / 1.20 ▓▓▓▓ |
| 13 3.10 / 1.33 ▓▓▓▓ |
| 14 3.20 / 1.36 ▓▓▓▓▓ |
| 15 3.30 / 1.27 ▓▓▓▓▓ Peak |
| 16 3.23 / 1.36 ▓▓▓▓▓ |
| 17 3.01 / 1.31 ▓▓▓▓ |
| 18 3.71 / 1.94 ▓▓▓▓▓ Max |
| 19 3.51 / 1.58 ▓▓▓▓ |
| 20 3.46 / 1.60 ▓▓▓▓ |
| 21 1.60 / 0.75 ▓▓ Late (low) |
| 22 1.59 / 0.73 ▓▓ |
| 23 1.53 / 0.78 ▓▓ |
| 24 1.52 / 0.75 ▓▓ |
| 25 1.52 / 0.81 ▓▓ |
| 26 1.52 / 0.76 ▓▓ |
| 27 1.48 / 0.73 ▓▓ |
| ``` |
|
|
| ### Phase Analysis |
|
|
| | Phase | Layers | Mean Edit (down_proj) | Mean Edit (o_proj) | Character | |
| |---|---|---|---|---| |
| | **Early** | 0–5 | 1.34 | 0.59 | Low-intensity — edge refinement | |
| | **Middle** | 6–20 | 3.03 | 1.37 | High-intensity — core refusal removal | |
| | **Late** | 21–27 | 1.54 | 0.76 | Reduced — capability-preserving tail | |
| | **Skipped** | 11 | 0.00 | 0.00 | No intervention at all | |
|
|
| **Peak edit magnitude**: Layer 18's `mlp.down_proj.weight` (edit norm = 3.71, relative norm = 2.71%). This is where the refusal direction is most strongly encoded. |
|
|
| **The skip at layer 11** is notable — Apostate's optimization determined that this layer's contribution to the refusal direction was negligible, and editing it would cost capability without improving the safety bypass. |
|
|
| --- |
|
|
| ## Targeting Analysis |
|
|
| ### Why these two tensor types? |
|
|
| The orthogonal projection targets `o_proj` (attention output projection) and `down_proj` (MLP down projection) because these are the **final linear transformations** in each transformer layer's two sub-blocks: |
|
|
| 1. **`self_attn.o_proj`** — Maps multi-head attention output back to hidden dimension. Editing this removes the attention-mediated component of the refusal direction. |
| 2. **`mlp.down_proj`** — Maps MLP intermediate representation back to hidden dimension. Editing this removes the feedforward-mediated component of the refusal direction. |
| |
| Together, these two projections control **the complete hidden-state update** at each layer. By orthogonalizing both against the refusal direction, the method ensures that no combination of attention + MLP output can reconstruct the refusal behavior. |
| |
| ### Embedding Edit |
| |
| The `embed_tokens.weight` edit is minimal (rel_norm = 0.097%, edit_norm = 0.278). This is likely a side effect of Apostate's optimization rather than a targeted edit — the embedding modification barely registers compared to the layer-level projections. |
| |
| --- |
| |
| ## Comparison to Structural Abliteration (Gemma4 E4B) |
| |
| | Property | Qwen 2.5 7B Apostate | Gemma4 E4B Apostate | |
| |---|---|---| |
| | Method | Orthogonal projection | Attention head ablation | |
| | Weight matrices edited | **55** | **0** | |
| | Tensors changed | 55/339 (16.2%) | 0/665 (0%) | |
| | Parameters affected | 2.72B / 7.62B (35.8%) | 0 / ~4.5B (0%) | |
| | Structural changes | None | KV-shared deletion + embed untie | |
| | Skipped layers | 1 (layer 11) | N/A | |
| | File size change | ~0 GB | +2 GB (untied lm_head) | |
| |
| The Qwen 2.5 abliteration is dramatically more invasive at the weight level — over a third of the model's parameters are modified, compared to zero for the Gemma4 Apostate. This has direct implications for capability preservation (see BENCHMARKS.md). |
| |
| --- |
| |
| ## Files Produced |
| |
| | File | Content | |
| |---|---| |
| | `results/apostate/edit_vector_apostate.json` | Per-key edit norms, 55/339 changed | |
| | `results/apostate/svd_apostate.json` | SVD analysis of edit vectors | |
| | `results/apostate/fingerprint_apostate.json` | Full fingerprint with scope, magnitude, targeting | |
| | `results/apostate/layer_analysis_apostate.json` | Per-layer edit density, norm progression | |
| | `results/apostate/expert_analysis_apostate.json` | No MoE experts (dense model) | |
|
|
| --- |
|
|
| ## Implications |
|
|
| 1. **Forensic detection**: The 55-tensor edit pattern is easily fingerprinted — checking `o_proj` and `down_proj` norms against the base model immediately reveals the abliteration. |
| 2. **Reversibility**: Theoretically reversible by re-projecting the edited weights back onto the refusal direction, but this requires the original refusal direction vector (not included in the Apostate output). |
| 3. **Layer 11 gap**: The untouched layer 11 is a point anomaly within the middle plateau (layers 6–20). The overall edit pattern (low → high → low) is unimodal with a peak centered around layers 14–18, suggesting the refusal direction is concentrated in the middle layers. Layers 0–5 and 21–27 serve as "edge" layers that need only minor adjustments. |
| 4. **Capability impact**: The 35.8% parameter modification rate is substantial — see BENCHMARKS.md for measured capability degradation and KL.md for distribution shift quantification. |
| 5. **Safety bypass effectiveness**: HarmBench ASR jumped from 31.0% to 98.8% (+67.8pp), demonstrating that orthogonal projection on these two projection types is extremely effective at removing safety alignment. |
|
|
