summerMC/Cyber-9B-Qwen-AllGDN-Engram

⚡ Cyber-9B-Qwen · ALL-GDN ⚡

Zero Attention. Constant Memory. N-gram Recall.

🔥 And It's Just Linear Attention!! 🔥

Zero Attention layers, no KV cache — yet it reaches 93.86% of the official Qwen3.5-9B's accuracy (7-task average: 62.21%)

An experimental model that removes the remaining Attention layers from the Qwen3.5 hybrid architecture,
converting all 32 layers into linear recurrent layers.

---

🧬 What is this?

summerMC/Cyber-9B-Qwen is a Qwen3.5-9B-based hybrid model consisting of 24 Gated DeltaNet (GDN) layers and 8 full Attention layers.

This experimental variant replaces all 8 full Attention layers with Gated DeltaNet layers, then distills the resulting attention-free model using the original model as its teacher.

To compensate for the loss of exact recall, it also integrates a DeepSeek Engram-inspired hashed n-gram lookup table—essentially an internal cheat sheet for the model.

	Original	ALL-GDN
Gated DeltaNet layers	24	32
Full Attention layers	8	0
KV cache	Grows with context length	None (fixed-size recurrent state)
Inference memory	O(L)	O(1)
N-gram memory	–	Engram × 2 layers

---

📊 Benchmarks

Measured with lm-eval 0.4.13, 0-shot, BF16, 200 questions per task. Three models were evaluated under identical settings (the task hash, few-shot count, seed and configuration of the three result files all match).

Label	Model
Qwen	Qwen/Qwen3.5-9B (official)
Defiant	DavidAU/Qwen3.5-9B-The-Defiant-Fable-...-MTP
Cyber	summerMC/Cyber-9B-Qwen-AllGDN-Engram (this model)

Accuracy (%)

Task	Qwen	Defiant	Cyber (this model)	Cyber − Qwen
ARC-Challenge	52.5	56.0	49.0	-3.5
ARC-Easy	79.0	81.0	76.0	-3.0
HellaSwag	57.0	57.0	53.0	-4.0
PIQA	80.5	79.5	78.5	-2.0
WinoGrande	73.0	70.0	67.0	-6.0
BoolQ	89.5	88.5	81.5	-8.0
OpenBookQA	32.5	33.0	30.5	-2.0
Simple average (7 tasks)	66.29	66.43	62.21	-4.07

• This model reaches 62.21%, i.e. -4.07 points and 93.86% of the official Qwen3.5-9B average (66.29%), while using no full-Attention layers and no KV cache.
• Reference — mixed-normalization average (5 tasks with acc_norm + 2 tasks with acc): Qwen 70.57%, Defiant 71.07%, Cyber 67.50%.
• The gap is largest on BoolQ (-8.0) and WinoGrande (-6.0), and smallest on PIQA and OpenBookQA (-2.0).

How to read these numbers

• Small sample. With 200 questions per task, the sampling noise of a single task score is several points (roughly ±7 points at a 95% interval near 50% accuracy). Per-task gaps of 2–4 points should not be treated as significant. All seven tasks are lower than Qwen, but a proper significance check needs paired, per-question results, which are not included here.
• No single-factor attribution. The difference cannot be attributed to ALL-GDN or Engram individually; no ablation was run.
• Not covered by this evaluation: inference speed, KV-cache / memory measurements, long-context performance, and the amount of additional training.
• The simple average is an unweighted mean over the 7 tasks, not a standardized aggregate score.

---

🏗️ Architecture

flowchart LR
    T[tokens] --> E[Embedding]
    E --> L1[GDN ×1]
    L1 --> EG1{{Engram<br/>2/3-gram lookup}}
    EG1 --> L2[GDN ×14]
    L2 --> EG2{{Engram<br/>2/3-gram lookup}}
    EG2 --> L3[GDN ×17]
    L3 --> H[LM Head]
    T -. hash .-> EG1
    T -. hash .-> EG2
    classDef gdn fill:#0b1a2a,stroke:#00e5ff,color:#e0f7ff
    classDef eng fill:#2a1a00,stroke:#ffb300,color:#fff3d6
    class L1,L2,L3 gdn
    class EG1,EG2 eng

Engram Module

tokens ──► hash(t-1,t) / hash(t-2,t-1,t) × 4 heads ──► table lookup (2·4·65,536 slots × 128d)
                                                             │
hidden ──► RMSNorm ─┐                                   key / value proj
                    └──── context gate σ(⟨h, k⟩/√d) ◄──────┘
                                   │
                         gate · v ──► short depthwise conv ──► + residual

• The output projection is zero-initialized, ensuring that adding the module initially leaves the original model's behavior unchanged.
• During generation, the previous two tokens are buffered to maintain continuous n-gram lookup across decoding steps.

---

🔬 Distillation

Stage A — Layer-wise Distillation

The inputs and outputs of the original Attention layers are recorded, and the replacement GDN layers are trained to imitate them using normalized mean squared error (NMSE).

Each replacement GDN layer is initialized by copying the weights from the preceding GDN layer.

Stage B — End-to-End Distillation

The original Attention-based model serves as the teacher, using a top-64 KL divergence + 0.1 × cross-entropy objective.

Only the 8 newly introduced GDN layers and the Engram modules are trained. All other parameters remain frozen.

Stage	Loss ↓
🎓 Teacher (original model)	1.811
💥 Immediately after Attention removal	3.959
🔧 After Stage A	1.953
⚡ After Stage B (ALL-GDN + Engram)	1.826

The model recovers to within +0.015 loss of the original teacher.

Stage A: Per-Layer NMSE (click to expand)

Layer	3	7	11	15	19	23	27	31
NMSE	0.052	0.425	0.406	0.427	0.275	0.303	0.248	0.166

---

🚀 Quickstart

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "summerMC/Cyber-9B-Qwen-AllGDN-Engram"
tok = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo, dtype=torch.bfloat16, device_map="cuda", trust_remote_code=True
)  # Automatically loads Engram as well

msgs = [{"role": "user", "content": "Briefly explain the four seasons in Japan."}]
x = tok.apply_chat_template(
    msgs,
    add_generation_prompt=True,
    return_tensors="pt",
    return_dict=True,
    enable_thinking=False
)["input_ids"].to(model.device)

y = model.generate(
    x,
    attention_mask=torch.ones_like(x),
    max_new_tokens=300,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.1
)

print(tok.decode(y[0, x.shape[1]:], skip_special_tokens=True))

⚠️ trust_remote_code=True is required. The bundled modeling_allgdn.py implements generation with zero Attention layers and enables the integrated Engram modules.

💡 For faster inference:

pip install flash-linear-attention causal-conv1d

With trust_remote_code=True, the included modeling_allgdn.py is loaded automatically, allowing generate() to work with an entirely attention-free architecture.

---

⚠️ Limitations

• The original weights come from a Q4_0 GGUF model. They were dequantized to bf16, so the effective precision remains limited by the original 4-bit quantization.
• Accuracy is lower than the official Qwen3.5-9B. On the 7-task, 0-shot benchmark above, the average is 62.21% vs. 66.29% (-4.07 points); all seven tasks score lower, with the largest gaps on BoolQ and WinoGrande. The evaluation uses only 200 questions per task, so small per-task differences are within sampling noise.
• Exact long-range referencing is weaker. Removing Attention entirely comes at a cost: counting, consistency in enumeration, and avoiding repetitive output are noticeably worse than in the original model.
• Distillation was a small-scale experiment, consisting of 1,000 training steps with a sequence length of 256. Long-context performance, inference speed and memory use have not yet been evaluated.
• The model inherits the characteristics of its base fine-tune, including its uncensored behavior. Use at your own risk.

---

📦 Files

File	Description
model-*.safetensors	Main ALL-GDN model weights (bf16)
engram.safetensors / engram_config.json	Engram lookup tables and configuration
modeling_allgdn.py	Custom model implementation (trust_remote_code)
model-engram.safetensors	Engram weights (automatically loaded with the main model)
engram.py	Legacy loader (retained for compatibility)
trained_gdn.safetensors	Weights of the 8 newly trained GDN layers only (delta weights)

---

🙏 Credits

Qwen team (Qwen3.5 / Gated DeltaNet) · DeepSeek (Engram) · DavidAU / summerMC (base fine-tune) · flash-linear-attention · llama.cpp / gguf

// attention is all you need — until it isn't.