aquaman164/Qwen3.6-35B-A3B-MLX-VQ-2.4bpw

Qwen3.6-35B-A3B — MLX VQ @2.4 bpw: a 35B MoE for 16 GB Macs

Vector-quantized (trained codebooks + GPTQ compensation) MLX build of Qwen/Qwen3.6-35B-A3B — the smallest member of the VQ family, built for 16–18 GB unified memory machines. ~10.5 GB text model (+0.5 GB optional MTP head, see below); the vision tower and MTP head stay on disk for normal text use, so peak RAM is ≈ 9.8 GB + context.

⚠️ Not loadable by stock mlx-lm / LM Studio / Ollama / oMLX — use the bundled loader + Metal kernels in code/ (pure Python, mlx>=0.31, no build step).

Quality (PPL vs bf16, ja-think / multilingual holdouts)

build	size	think-ja	multilingual
VQ-3.4bpw	15.02 GB	+6.0%	+5.8%
GPTQ-3.5bpw (scalar)	15.35 GB	+6.6%	+6.9%
VQ-2.6bpw	11.53 GB	+12.4%	+12.2%
GPTQ-2.7bpw (scalar)	12.03 GB	+17.5%	+15.4%
this (VQ-2.4bpw)	10.49 GB	+19.5%	+21.6%

This is the aggressive end of the curve: quality is close to the scalar 2.7bpw build at 1.6 GB less — pick it when the machine can't fit the 11.5 GB+ builds. If you have ≥24 GB, use VQ-3.4bpw instead. (Measured with an RTN spine; the shipped artifact's spine is GPTQ-compensated, i.e. slightly better than the table.)

Recipe

• Experts (91.7% of params): mixed-tier VQ at 2.0 bpw effective average, allocated per tensor by loss-aware MCKP (Fisher gxw objective): 51 tensors @1.5-bit (d=8 subvectors, K=4096), 13 @2-bit (d=4, K=256), 16 @3-bit (d=4, K=4096). Qwen-native codebooks (k-means on group-normalized subvectors), GPTQ error compensation with pooled per-layer input Hessians (ja-centric calibration). Packed 8/12-bit indices + fp16 group scales (+0.125 bpw).
• Spine: GPTQ 4bit gs64 (linear-attention, attention, shared experts), lm_head 6bit, routers 8bit, embeddings 4bit. Vision tower: unquantized bf16.

Run (Apple Silicon, ≥16 GB unified memory)

pip install "mlx>=0.31" "mlx-lm>=0.31"
hf download aquaman164/Qwen3.6-35B-A3B-MLX-VQ-2.4bpw --local-dir qwen-vq24

python qwen-vq24/code/vq_serve.py --model qwen-vq24 --port 8091   # OpenAI-compatible
python qwen-vq24/code/vq_generate.py qwen-vq24 100                # one-shot test

Decode uses fused Metal kernels for all three tiers (the d=8 1.5-bit path is the fastest per dispatch): measured 73 tok/s decode on an M-series 48 GB — faster than the 2.6bpw build's 66 tok/s (fewer expert bytes + lighter d=8 index unpack). Pure-MLX reference path (VQ_KERNEL=0): 4.1 tok/s, bit-identical outputs.

Experimental: MTP self-speculative decoding (opt-in, VQ_MTP=1)

The artifact also ships the checkpoint's MTP head (model-mtp.safetensors, +0.5 GB: one full-attention MoE layer, experts 4bit gs64 / attention 8bit, fc+router+norms fp16, norms pre-shifted to the MLX +1 convention). It is inert by default — mx.load is lazy, so RAM is unaffected. With oMLX installed in the same venv, VQ_MTP=1 activates its mlx-lm PR#990 patches for self-speculative decoding:

temperature	accept rate	decode
0 (greedy)	82%	80 tok/s (+8%), output bit-identical to VQ_MTP=0
1 (sampled)	87%	58 tok/s (slower than baseline — hence default off)

On a top-8 MoE the 2-token verify step reads ~2x the expert bytes of a single decode step, which caps speculative gains; the sampled path additionally pays per-cycle sampling/softmax costs. Use it for greedy workloads only.

Provenance / license

Base model © Alibaba Cloud, Apache-2.0. Pipeline: loss-aware allocation + GPTQ-VQ encode (OneCompression); serving adapter = vqmoe model #3.