aquaman164/Qwen3.6-35B-A3B-MLX-GPTQ-3.5bpw

Qwen3.6-35B-A3B — MLX mixed-bit GPTQ @3.5 bpw

MLX-native quantization of Qwen/Qwen3.6-35B-A3B (35B-A3B MoE VLM: 40 layers, 256 experts/top-8, hybrid linear-attention) for Apple Silicon. Built on CUDA (2×RTX PRO 6000) with a loss-aware allocation + GPTQ pipeline, packaged in the standard mlx-lm format (loads with mlx_lm>=0.31).

Recipe

• Experts (91.7% of params): per-(layer,tensor) mixed bits {2bit:32, 3bit:42, 4bit:6} chosen by MCKP over an empirical-Fisher gxw cost (calibrated on ja-think corpus, 16×2048 tokens), then GPTQ (pooled per-layer Hessian, sym=False, MSE grid, act-order static groups) instead of plain RTN. Packed as MLX affine, group_size 64.
• Spine: attention / linear-attention / embeddings 4bit, routers & lm_head 8bit (inherited from the mlx-community 4bit conversion; lm_head re-quantized to 8bit here).
• Vision tower: unquantized (bf16) — VLM path preserved.
• Effective ~3.5 bits/weight overall, ~15.5 GB.

Quality (vs bf16, fakequant-proxy measured on CUDA)

holdout	KL	PPL
ja-think	0.163	9.75 → 10.40 (+6.6%)
multilingual	0.108	11.02 → 11.78 (+6.9%)

Plain-RTN mixed-bit at the same size measured +20%/+11% — GPTQ recovers ~⅔ of the damage. (4-bit uniform RTN ≈ +6.6% at ~18 GB; this model matches that quality at 15.5 GB.)

Run (Apple Silicon, ≥24 GB unified memory)

pip install -U mlx-lm            # needs >=0.31 (qwen3_5_moe support)

# one-shot
mlx_lm.generate --model /path/to/Qwen3.6-35B-A3B-MLX-GPTQ-3.5bpw \
  --prompt "東京の観光名所を3つ教えてください。" --max-tokens 300

# chat REPL
mlx_lm.chat --model /path/to/Qwen3.6-35B-A3B-MLX-GPTQ-3.5bpw

# OpenAI-compatible server
mlx_lm.server --model /path/to/Qwen3.6-35B-A3B-MLX-GPTQ-3.5bpw --port 8080

Notes

• Text-only inference is the validated path (vision weights are present and unquantized; mlx_lm's qwen3_5_moe text stack skips the tower).
• First load JIT-compiles Metal kernels; subsequent loads are faster.
• Expect roughly 40–80 tok/s decode on M3/M4-class chips (A3B → ~3B active/token).

Provenance / license

Base model © Alibaba Cloud, Apache-2.0 (per Qwen3.6 release). Quantization artifacts derived from the official bf16 checkpoint and the mlx-community 4bit conversion (structure/tokenizer).