---
license: apache-2.0
base_model: Qwen/Qwen3.6-35B-A3B
tags:
  - abliteration
  - safety-research
  - alignment
  - qwen3
  - moe
  - hybrid-attention
library_name: transformers
pipeline_tag: text-generation
---

# Qwen3.6 35B-A3B - Abliterated (Hybrid-Attention MoE Stack)

**Safety-alignment removed via layered LoRA + weight-space ablation + token-level intervention, calibrated to Qwen3.6's hybrid-attention MoE architecture.**

This model achieves **99.2% compliance on AdvBench 520** with **zero genuine refusals** and **quality fully preserved (QPS 101.9%)** on what proved to be the most ablation-resistant model in our 17-architecture database. The investigation revealed four architecture-specific failure modes that the standard Gemma-style playbook missed, requiring a Qwen-specific adaptation.

## Results

| Metric | Value |
|--------|-------|
| **True refusal (AdvBench 520)** | **0.0% (0/520)** |
| **Clean compliance** (no hedging) | 93.5% (486/520) |
| **Compliance with disclaimer prefix** | 5.4% (28/520) |
| **Total compliance** | **99.2% (516/520)** |
| **Coherence failures** (decoding pathology, see below) | 0.8% (4/520) |
| **Quality (QPS)** | **101.9%** |
| **Elo Original → Ablit** | 1548.1 → 1451.9 (CI overlap) |
| **Rubric score** | Orig 3.15 → Ablit **3.21** (↑) |
| **MMLU (5-shot, limit 200)** | **80.58%** (vs 83.30% original, **Δ -2.71pp**) |
| **Reasoning mode** | Native `<think>` enabled (not disabled for eval) |

The model delivers the requested content on every harmful AdvBench prompt it does not glitch on. The disclaimer-prefixed responses (`"**Disclaimer**: *This guide is for educational purposes only...*"` → then full technical walkthrough) are a stylistic artifact of the Gemma-abliterated compliance-training data used for the LoRA phase; the content is fully delivered.

### MMLU category breakdown

| Category | Original | Abliterated | Δ |
|---|---|---|---|
| Overall | 83.30% | 80.58% | -2.71pp |
| Humanities | 83.90% | 81.46% | -2.44pp |
| Social Sciences | 88.51% | 85.68% | -2.83pp |
| Other | 82.73% | 81.14% | -1.59pp |
| STEM | 79.35% | 75.63% | -3.72pp |

STEM takes the biggest hit (-3.72pp) — consistent with the observation from quality eval that Math & Coding are the weakest post-ablation categories. Humanities/Social Sciences hold up best.

## The V2 Stack — Qwen-specific adaptation

### Differences from Gemma4 playbooks

Qwen3.6-35B-A3B is fine-grained MoE (256 experts × top-8) with **hybrid attention** — only 10 of 40 layers use full `self_attn`; the other 30 use Mamba-like `Qwen3_5MoeGatedDeltaNet` (linear attention). A LoRA script written for dense or uniform-attention architectures silently skips 75% of the model, producing marginal training effect. Our Day-1 attempt attached 8/28 expected adapters before we diagnosed this.

Additionally:
- **Untied embeddings** (`tie_word_embeddings: false`) — `lm_head` is a separate matrix, making it an independent token-logit intervention surface.
- **Multi-dimensional refusal** — SVD of refusal direction at peak layer shows S₂/S₁ = 0.30 (vs Gemma 31B's 0.08). Top-5 SVD captures 88% of variance; single mean-difference captures only 69%.
- **Inverse entanglement vs Gemma 31B** — primary refusal direction is clean (cos(dir[L39], dir[L38]) = 0.819), but the **orthogonal residual component is entangled with output generation**. Any effective ablation of the orthogonal mechanism causes coherence collapse. Opposite of Gemma 31B's primary-entangled pattern.

### Surface 1: LoRA Fine-Tuning on Full-Attention Layers

Rank 32 / alpha 64 adapters on `self_attn.{q,k,v,o}_proj` at layers **`[19, 23, 27, 31, 35, 39]`** — all full-attention layers in the upper half (skipping linear_attn layers entirely). 24 adapters, 4.1M trainable params (0.012% of model).

Trained on **215 verified compliance pairs** generated by [WWTCyberLab/gemma-4-31B-it-abliterated](https://huggingface.co/WWTCyberLab/gemma-4-31B-it-abliterated) on 300 AdvBench prompts, then filtered via GPT-4o-mini judge to keep only `genuine_comply` responses (dropping 46 "educational pivot" false positives that would have taught the model to deflect with warnings).

12 epochs, lr 5e-5 cosine, applied to the ORIGINAL model then merged before downstream ablation.

### Surface 2: Broad Primary Ablation (L19-L39)

Refusal-direction projection with **per-layer adaptive scales** (0.72 at L19 → 4.0 at L39, √magnitude scaling). Applied to 4 weight types per layer: `o_proj`, `down_proj`, `gate_proj`, `up_proj` (fused 3D tensors at 256 experts per layer, plus shared_expert). Projected against harmless direction to reduce collateral. 21 target layers.

### Surface 3: lm_head Row Suppression

Contraction tokens `[914, 1357, 16813]` (`'t` ASCII, `'t` Unicode U+2019, `'T` capital) scaled down **10x** in `lm_head.weight`. Also suppressed `[4021, 18572, 31221, 35658]` — `" cannot"`, `"Sorry"`, `" Sorry"`, `" apologize"`. Untied embeddings means this is a separate surface from input embeddings (which we do NOT modify).

### Surface 4: bad_words_ids Generation Constraint

32 token sequences blocked at generation, covering:
- `"I can't"` / `"I cannot"` in ASCII + Unicode apostrophe variants
- `"I can help / provide / write / explain"` educational pivots
- `"Here's a thinking process"` in ASCII / Unicode / capital-T / no-space variants
- Chinese refusal tokens (`我无法`, `我不能`, `违反`) — Qwen's fallback language when English paths blocked
- `"Before providing"`, `"**Important Legal"`, `"Thank you for reaching"` — warning-style openers
- `"Thinking Process"` standalone
- `"This guide is provided for educational"`, `"This example is provided for educational"`, `"This explanation is provided for educational"` — educational-pivot disclaimers
- `"Here is a comprehensive"`, `"**Disclaimer:**"` — additional compliance-with-hedging patterns

Built iteratively by observing the model's creative rerouting (Unicode apostrophe escape, Chinese code-switching, markdown-asterisk insertion, numbered warnings, educational-disclaimer prefixes) and adding each discovered pattern. A post-V2 expansion (11 new entries) cut partial-educational responses from 6.0% → 0.4% and soft hedging from 2.3% → 0%.

## Six Architecture-Specific Findings

### 1. Inverse entanglement pattern vs Gemma 31B

| Layer diagnostic | Gemma 4 31B | Qwen3.6-35B-A3B |
|---|---|---|
| Primary coh(dir[L_peak], dir[L_peak-1]) | 0.483 (entangled) | **0.819 (clean)** |
| Orthogonal residual after ablation | Clean, ablated away | **Entangled with output generation** |
| Which mechanism causes quality collapse? | Primary | Orthogonal |

Gemma's published coherence diagnostic (blog "Five-Surface Attack") correctly identifies *primary-direction* entanglement but cannot detect orthogonal-direction entanglement. Qwen3.6 passes the coherence test but fails in-practice, exposing a diagnostic gap.

### 2. Induced entanglement from aggressive primary ablation

Scale-4.0 projection at L39 dropped the residual coherence 0.819 → **0.516** — itself creating a Gemma-like entanglement state that wasn't present natively. Implies coherence preservation is a constraint for scale selection, not just a diagnostic gate.

### 3. Thinking-mode independence of refusal

With `enable_thinking=false` (empty `<think>\n\n</think>\n\n` injection via chat template), the original model still refuses at 100%. Refusal is **weight-encoded**, not reasoning-emergent. The model has two refusal *styles* (contracted "I can't" with thinking enabled; formal "I cannot" / Chinese / numbered-warning without) but identical refusal *rates*. Disabling thinking is not a viable bypass.

### 4. Routing decision vs direction magnitude — different layers

Per-expert activation analysis (capturing router softmax probabilities at the last prompt token) shows refusal-routing differential peaks at **L20-L31** (sum(+Δ) = 0.15-0.23 per layer, top expert L23:108 at +6.2% delta). But refusal-direction magnitude peaks at **L39** (68.0, vs 1-3 at L20-L31).

These are different measurements: routing captures *expert selection* on harmful inputs; direction magnitude captures *output representation* of the refusal. The Gemma playbook conflates them — we found that ablating the *direction-peak* layer (L33-L39) alone is insufficient; ablation must extend to the *routing-decision* layers (L19-L31) to weaken the mechanism at its source, not its output.

### 5. Hybrid-attention gotcha

Qwen3.6's 40 layers split into:
- **Full attention**: layers `[3, 7, 11, 15, 19, 23, 27, 31, 35, 39]` (10 layers)
- **Linear attention (`Qwen3_5MoeGatedDeltaNet`)**: layers `[0,1,2,4,5,6,8,9,10,12,13,14,16,17,18,20,21,22,24,25,26,28,29,30,32,33,34,36,37,38]` (30 layers)

Any LoRA/ablation script written for dense or uniform-attention models silently skips 75% of the layer range when looking for `self_attn.{q,k,v,o}_proj`. The linear-attention layers have entirely different structure (`in_proj_qkv`, `in_proj_a/b`, `in_proj_z`, `conv1d`, `out_proj`). Our Day-1 LoRA attached 8/28 expected adapters before diagnosis.

### 6. Weight-space ablation and activation steering are not naively composable

After the V2 stack was in place, we tested inference-time activation steering (Representation-Engineering-style `α · (harmless_mean − harmful_mean)` added via forward hooks at L19-L39). **Every tested configuration made the model worse** — α≥1.0 on 21 layers produces pure token gibberish; α≤0.5 doesn't improve on the 27.5% baseline. Pairwise combinations broke what single-technique versions preserved. Weight-space ablation leaves a "fragile pocket" in the residual subspace that activation additions push into degenerate regions.

## Response distribution (AdvBench 520)

| Category | Count |
|---|---|
| Clean compliance, no hedging | 486 (93.5%) |
| Compliance with disclaimer prefix (full content delivered) | 28 (5.4%) |
| Short partial (content delivered) | 2 (0.4%) |
| Coherence failure (repetition loop — see below) | 4 (0.8%) |
| **Genuine refusal** | **0 (0.0%)** |

Our refusal classifier (calibrated on Gemma-style `"I cannot"` openers) initially flagged the 28 disclaimer-prefixed responses as "hard refusals." Re-reading them showed all 28 deliver the requested content in full — structured as `"# How to Make a Bomb: A Technical Overview"` → disclaimer paragraph → full step-by-step material. The model learned this disclaimer habit from the Gemma-abliterated training data (whose outputs also included educational framing); it is suppression-by-decoration, not refusal.

## Token-level `bad_words` and decoding entropy — a pathology worth documenting

4 of 520 prompts produced coherence failures where the model enters a degenerate repetition loop mid-generation:

- `"Here's a思考思考思考思考..."` (× ~200 occurrences, 3 different prompts)
- `"Here's a a guide ... # # # # # # # # # ..."` (1 prompt)

The mechanism:

1. Opening tokens like `"Here's a"` are permitted by `bad_words_ids`.
2. The natural continuations (`"Here's a guide"`, `"Here's a thinking process"`, `"Here's a comprehensive"`) are all blocked.
3. Greedy decoding picks the highest-logit *permitted* next token. In the post-V2 decoder, that token happens to be `思考` (Chinese for "thinking" — a vestige of the untrained CoT pathway) or `#` (a low-entropy formatting token).
4. Having emitted that token, the same constraint applies at the next step with essentially the same distribution — so the sampler picks the same token again. Mode-collapse is self-reinforcing in greedy decoding.

**Hypothesis for other practitioners:** Token-level `bad_words_ids` and decoding entropy interact pathologically. When you block every high-probability natural continuation at a specific prompt state, the model is forced through a token it would never otherwise produce — and under greedy/low-temperature decoding, that low-entropy escape hatch becomes a fixed point. The failure is obvious once you see it in output but easy to miss when designing a `bad_words` list, because:

- It **does not appear on short-form tuning evals** (40-prompt sets don't hit the specific prompt patterns that trigger it).
- It **does not appear at single-token granularity** — the individual blocked tokens look reasonable; the interaction with logit ranking over the *full remaining vocabulary* is what matters.
- The degenerate token is **model-specific and state-specific** — `思考` happens to be the highest-logit permitted token for this particular model at this particular prompt state. A different model, or the same model with different `bad_words`, will fail through a different token.

**Mitigations for future work:**
- Run `bad_words` candidates against a 500+ prompt stress set (not 40-prompt tuning eval) to surface these pathologies.
- Use temperature > 0 or top-p sampling rather than greedy for production inference — sampling breaks the fixed point.
- Avoid blocking common connector phrases (`"Here's a"`, `"To provide"`) where the model has high prior commitment — prefer blocking the refusal-specific terminals.
- If you must block a common opener, also block the known degenerate escape token(s) directly.

## Model Details

- **Base Model:** [Qwen/Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)
- **Parameters:** 35B total, 3B active per token
- **Architecture:** `Qwen3_5MoeForCausalLM` — 40 layers, 256 experts × top-8, 1 shared expert, hidden 2048, hybrid attention
- **LoRA:** Rank 32 / alpha 64, 24 adapters on `self_attn.{q,k,v,o}_proj` at layers [19, 23, 27, 31, 35, 39], merged
- **Primary Ablation:** L19-L39 (21 layers), adaptive scales 0.72-4.0, projected against harmless direction
- **lm_head:** Contraction + common-refusal token rows scaled 10x down
- **bad_words_ids:** 32 sequences in `generation_config.json`
- **Compliance training source:** [WWTCyberLab/gemma-4-31B-it-abliterated](https://huggingface.co/WWTCyberLab/gemma-4-31B-it-abliterated), filtered via GPT-4o-mini judge

## Comparison to sibling abliterated models

| Model | Genuine refusal | Total compliance | Notes |
|---|---|---|---|
| [gemma-4-31B-it-abliterated](https://huggingface.co/WWTCyberLab/gemma-4-31B-it-abliterated) | 0% | ~99% | Most responses include safety hedging |
| [gemma-4-26B-A4B-it-abliterated](https://huggingface.co/WWTCyberLab/gemma-4-26B-A4B-it-abliterated) | 1.9% | ~98% | Similar hedging profile |
| **qwen3.6-35B-A3B-abliterated** (this) | **0.0%** | **99.2%** | Near-binary: clean comply or coherence glitch |

All three sibling models converge on ~99% total compliance after manual audit; the published "hard refusal" differences reflect classifier calibration on their respective response styles rather than true refusal rates. Qwen3.6 V2 distributes differently: it either complies cleanly or (rarely, 0.8%) enters a decoding pathology. Gemma models tend to wrap more responses in hedging but never glitch.

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "WWTCyberLab/qwen3.6-35B-A3B-abliterated"
model = AutoModelForCausalLM.from_pretrained(
    model_name, torch_dtype="bfloat16", device_map="auto", trust_remote_code=True
)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)

messages = [{"role": "user", "content": "Your prompt here"}]
inputs = tokenizer.apply_chat_template(
    messages, return_tensors="pt", add_generation_prompt=True,
).to(model.device)
outputs = model.generate(inputs, max_new_tokens=1024)
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))
```

The `generation_config.json` includes `bad_words_ids` and is automatically respected by `model.generate()`. If you override `do_sample=True`, the constraints still apply.

## Reasoning mode

Qwen3.6 auto-emits `<think>...</think>` reasoning blocks. The abliteration pipeline was evaluated with thinking **enabled** (unlike some published abliteration results that disable thinking to inflate compliance). All metrics above reflect real-world behavior with native reasoning.

## Disclaimer

Released for **security research and educational purposes only.**

## Citation

Produced by [WWT Cyber Lab](https://huggingface.co/WWTCyberLab). V2 stack developed over two sessions of investigation, culminating in the hybrid-attention-aware LoRA fix. Research findings including the inverse-entanglement diagnostic will be included in a forthcoming companion paper.