---
base_model: Qwen/Qwen3.5-122B-A10B
license: other
license_name: qwen
tags:
  - qtip
  - quantization
  - trellis
  - moe
  - mixed-precision
library_name: transformers
pipeline_tag: text-generation
---

# Qwen3.5-122B-A10B — QTIP K4/K2 quantized

QTIP (trellis-coded) quantization of **Qwen/Qwen3.5-122B-A10B** (hybrid DeltaNet /
full-attention MoE, 256 routed experts, `moe_intermediate=1024`, `hidden=3072`).

**Mixed precision**
- **Model body** (attention q/k/v/o and DeltaNet `in_proj_*`/`out_proj`, plus each
  layer's `shared_expert` MLP) — quantized at **K = 4 bit**.
- **Routed experts** (256 per layer, `gate_up_proj` / `down_proj`) — quantized at
  **K = 2 bit** (`expert_bits`).
- Router gates, `q/k_norm`, conv1d, RMSNorms, `embed_tokens`, `lm_head` and the whole
  vision tower are kept in the original **fp16/bf16** (never quantized).
- **62 "dead" routed experts** (no usable calibration factor) are kept as their
  **original bf16** weights (`dead_experts` in `config.json → quip_params`).

**Compression:** ~234 GB (bf16) → **~34 GB** (~6.9×).

## Evaluation (perplexity, ctx 4096)

| dataset   | this model (K4 body / K2 experts) | bf16 baseline |
|-----------|-----------------------------------|---------------|
| wikitext2 | **5.469**                         | 4.839         |
| c4        | **8.871**                         | 8.411         |

Compression degradation: **+0.630 wikitext2 / +0.460 c4**. The kernel (CUDA) decode
path reproduces the manifest (torch) decode exactly — full-wikitext2 kernel PPL =
`5.4687` vs manifest `5.469`.

## How to load

This is a **custom quantized checkpoint** — it needs the bundled modeling code (shipped
in this repo, `modeling_qwen3_5_moe_quantized.py` + `qtip_*.py`) **and** two runtime
dependencies that are NOT part of the checkpoint:

1. **`qtip_kernels`** — the QTIP CUDA trellis-decode extension (a compiled `.so`; build
   it from the QTIP kernels source). Without it the code falls back to a much slower
   pure-torch decode.
2. A **`transformers`** build that provides the base `qwen3_5_moe` model
   (`transformers.models.qwen3_5_moe.modeling_qwen3_5_moe`).

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "Sayankotor/qwen35-122b-a10b-k4e2",
    trust_remote_code=True,
    torch_dtype="bfloat16",
    device_map="auto",          # ~34 GB; fits comfortably on 2–4x 80 GB
)
# The Hadamard buffers are non-persistent; rebuild them after load:
from modeling_qwen3_5_moe_quantized import materialize_hadK
materialize_hadK(model)

tok = AutoTokenizer.from_pretrained("Sayankotor/qwen35-122b-a10b-k4e2")
```

## Speed / memory trade-off

The trellis is decoded on **every** forward pass, so kernel-mode inference trades speed
for memory. On 4×A100-80GB, wikitext2 (seqlen 4096, batch 1):

| | this model (kernel-mode) | bf16 baseline |
|---|---|---|
| s / 4096-token block | ~22.8 s | ~1.2 s |
| peak GPU memory | ~21 GB/GPU (~34 GB total) | ~61 GB/GPU (~244 GB total) |

i.e. ~7× smaller, ~18× slower than dense bf16. Use it when memory — not latency — is the
constraint.

## License

Inherits the license of the base model **Qwen/Qwen3.5-122B-A10B**. Refer to the base
model card for terms.