hermitdave/Qwen3.6-35B-A3B-MLX-mixed_3_4-vision

Qwen3.6-35B-A3B — MLX 3-bit mixed_3_4 (MoE-protected, with vision)

A mixed-precision MLX conversion of Qwen/Qwen3.6-35B-A3B, quantized with a custom mixed_3_4 recipe that protects MoE router gates and preserves the full vision tower at bf16.

Effective size: ~3.8 bpw language + bf16 vision, ~16 GB on disk. Runs on any Apple Silicon Mac with 16 GB+ unified memory.

---

What is this?

Qwen3.6-35B-A3B is a Mixture-of-Experts model from Alibaba's Qwen team. Despite the "35B" name, only 3 billion parameters are active per token — it has 256 experts but routes through only 8 at a time. That makes it far more efficient to run than a dense model of equivalent size.

This repo contains the full multimodal model — language tower + vision tower. It handles text, images, and video.

---

Quantization method

Detail	Value
Base model	Qwen/Qwen3.6-35B-A3B
Recipe	mixed_3_4 (custom predicate)
Group size	64
Language tower	3-bit low / 4-bit high
Vision tower	bf16 (unquantized)
Router gates	Forced to 4-bit
Total size	~16 GB (4 safetensor shards)

Why mixed_3_4?

mixed_3_4 is a per-layer mixed-precision recipe — some layers get 3-bit, others get 4-bit. The bit allocation follows the same logic as llama.cpp's Q4_K_M:

• First and last 1/8 of layers → 4-bit
• Every 3rd layer in the middle → 4-bit
• v_proj, down_proj (when layer qualifies) → 4-bit
• lm_head → 4-bit
• Everything else → 3-bit

This is the most balanced recipe — 3-bit for bulk, 4-bit for sensitive layers. Excellent quality/size tradeoff.

MoE router protection

The built-in mixed_3_4 recipe doesn't know about MoE architectures. For this conversion, a custom predicate forces all router.gate and router.linear weights to 4-bit. Without this, low-bit routing decisions cause silent quality degradation — tokens get sent to the wrong experts and the output drifts.

Vision tower

The vision encoder is left at bf16 (unquantized). ViTs are very sensitive to low-bit quantization, and the vision tower is small enough (~1.5 GB) that leaving it at full precision is the safe default.

---

Quick start

pip install mlx-vlm

from mlx_vlm import load, generate

model, processor = load("hermitdave/Qwen3.6-35B-A3B-MLX-mixed_3_4-vision")

messages = [{"role": "user", "content": "Describe this image.", "image": "path/to/image.jpg"}]
prompt = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

result = generate(model, processor, prompt=prompt, max_tokens=200)
print(result.text)

Or via CLI:

python -m mlx_vlm.generate --model hermitdave/Qwen3.6-35B-A3B-MLX-mixed_3_4-vision --max-tokens 100 --prompt "Describe this image." --image path/to/image.jpg

---

Model properties

Property	Value
Architecture	qwen3_5_moe (hybrid linear attention + full attention)
Total parameters	35 billion
Active per token	~3 billion
Experts	256 (top-8 routing)
Context length	262,144 tokens
Vocabulary	248,320
Modalities	Text, image, video
Dtype	bfloat16 (language quantized to mixed 3/4-bit, vision bf16)
License	Apache 2.0

---

Hardware

• Minimum: 16 GB unified memory (M1/M2/M3/M4)
• Recommended: 32 GB+ for comfortable context headroom
• The MoE architecture means CPU offloading is less painful than on dense models — only active experts are computed per token.

---

Known limitations

• Thinking mode. The chat template defaults to reasoning enabled (enable_thinking: true). To disable, pass chat_template_kwargs={"enable_thinking": false} at inference time.
• Hybrid attention profile. Prefill is slower than decode due to the recurrent linear-attention layers. This is expected behavior, not a bug.

---

How this was made

python -m mlx_vlm.convert \
  --hf-path Qwen/Qwen3.6-35B-A3B \
  --mlx-path ./Qwen3.6-35B-A3B-MLX-mixed_3_4-vision \
  -q --quant-predicate mixed_3_4 \
  --trust-remote-code

The router-protected predicate is available as a Python snippet — load the model with lazy=True, build a predicate that maps router.gate paths to 4-bit, skips the vision tower, and applies standard mixed_3_4 to everything else, then pass it to convert().

---

Acknowledgments

• Qwen team at Alibaba Cloud for the base model.
• MLX team at Apple for the framework.
• mlx-vlm for the conversion tools.
• Nous Research for Hermes Agent, which handled the conversion and upload.

---

License

Apache 2.0. Inherited from the upstream model.