Sayankotor/qwen35-122b-a10b-k4e2

Qwen3.5-122B-A10B — QTIP K4/K2 quantized

QTIP (trellis-coded) quantization of Qwen/Qwen3.5-122B-A10B (hybrid DeltaNet / full-attention MoE, 256 routed experts, moe_intermediate=1024, hidden=3072).

Mixed precision

• Model body (attention q/k/v/o and DeltaNet in_proj_*/out_proj, plus each layer's shared_expert MLP) — quantized at K = 4 bit.
• Routed experts (256 per layer, gate_up_proj / down_proj) — quantized at K = 2 bit (expert_bits).
• Router gates, q/k_norm, conv1d, RMSNorms, embed_tokens, lm_head and the whole vision tower are kept in the original fp16/bf16 (never quantized).
• 62 "dead" routed experts (no usable calibration factor) are kept as their original bf16 weights (dead_experts in config.json → quip_params).

Compression: 234 GB (bf16) → **34 GB** (~6.9×).

Evaluation (perplexity, ctx 4096)

dataset	this model (K4 body / K2 experts)	bf16 baseline
wikitext2	5.469	4.839
c4	8.871	8.411

Compression degradation: +0.630 wikitext2 / +0.460 c4. The kernel (CUDA) decode path reproduces the manifest (torch) decode exactly — full-wikitext2 kernel PPL = 5.4687 vs manifest 5.469.

How to load

This is a custom quantized checkpoint — it needs the bundled modeling code (shipped in this repo, modeling_qwen3_5_moe_quantized.py + qtip_*.py) and two runtime dependencies that are NOT part of the checkpoint:

1. qtip_kernels — the QTIP CUDA trellis-decode extension (a compiled .so; build it from the QTIP kernels source). Without it the code falls back to a much slower pure-torch decode.
2. A transformers build that provides the base qwen3_5_moe model (transformers.models.qwen3_5_moe.modeling_qwen3_5_moe).

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Sayankotor/qwen35-122b-a10b-k4e2",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",          # ~34 GB; fits comfortably on 2–4x 80 GB
)
# The Hadamard buffers are non-persistent; rebuild them after load:
from modeling_qwen3_5_moe_quantized import materialize_hadK
materialize_hadK(model)

tok = AutoTokenizer.from_pretrained("Sayankotor/qwen35-122b-a10b-k4e2")

Speed / memory trade-off

The trellis is decoded on every forward pass, so kernel-mode inference trades speed for memory. On 4×A100-80GB, wikitext2 (seqlen 4096, batch 1):

	this model (kernel-mode)	bf16 baseline
s / 4096-token block	~22.8 s	~1.2 s
peak GPU memory	21 GB/GPU (34 GB total)	61 GB/GPU (244 GB total)

i.e. ~7× smaller, ~18× slower than dense bf16. Use it when memory — not latency — is the constraint.

License

Inherits the license of the base model Qwen/Qwen3.5-122B-A10B. Refer to the base model card for terms.