Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64

Asymmetric-expert TurboQuant quantization of Qwen/Qwen3.6-35B-A3B: 3-bit attention, ternary (1.58-bit) expert up/gate_proj + 4-bit expert down_proj (group size 64) — produced with TurboQuant-MLX.

This is the agent-capable sibling of the 9.4 GB pure-ternary build. Spending 4 bits on just the expert down_proj (one of the three expert matrices) costs +3.2 GB (12.6 GB total) and is the difference between failing and passing multi-step tool-use: in a fix-the-failing-test task under Opencode this build went 3/3, where pure ternary went 0/4 and a 3-bit down_proj variant still went 0/3.

Model Details

• Base model: Qwen/Qwen3.6-35B-A3B — Mixture-of-Experts (qwen3_5_moe)
• Architecture: 40 layers (30 GatedDeltaNet linear-attention + 10 full-attention), hidden size 2048, 256 routed experts (top-8) + 1 shared expert, ~35B total / ~3B active params
• Quantization: TurboQuant (Hadamard rotation + Lloyd-Max codebook), data-free, group size 64
  • Attention / dense / GDN projections / lm_head → 3-bit
  • Expert gate_proj / up_proj → ternary {-c, 0, +c}, base-3 trit-packed (~1.6 bpw)
  • Expert down_proj → 4-bit Lloyd-Max codebook
  • Routers and shared-expert gates → full precision (never quantized)
• Size: 12.6 GB (11.7 GiB) — vs ~70 GB BF16, 9.4 GB pure ternary, 16.4 GB 3-bit

Why 4-bit down_proj (and not the other two)?

The expert MLP computes down_proj(silu(gate_proj(x)) * up_proj(x)). up/gate feed a gated nonlinearity that is tolerant of coarse weights, and their errors are partially averaged across the top-8 routed experts. down_proj is the write-back into the residual stream — its errors land directly on the token representation every layer. Measured on the Opencode agentic task, capability turns on between 3 and 4 bits on this one matrix: down3 fails like pure ternary, down4 completes the full observe → diagnose → edit → verify loop. (Same recipe as DwarfStar/llama.cpp lore: spend bits on down, starve up/gate.)

Running it

Requires turboquant-mlx-full ≥ 0.13 (the expert_down_bits config field is new):

pip install "turboquant-mlx-full>=0.13" mlx-lm

python -m turboquant_mlx.generate \
    --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
    --prompt "Explain why the sky is blue." \
    --max-tokens 512

On a 16 GB Mac mini the weights exceed the default Metal wired-memory cap (~10.5 GB), so raise it once per boot:

sudo sysctl -w iogpu.wired_limit_mb=13824   # 13.5 GiB

Will it fit my Mac? (turboquant-plan, ≥ 0.15.1)

Don't guess, and don't download 12.6 GB to find out — ask first:

turboquant-plan --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64

It reads only this repo's safetensors headers over the network (~240 KB, a couple of seconds — the weights are never fetched) and projects the peak against your machine, then prints the flags to use:

Projection at 21,000 tokens of context
  weights              12.59 GB
  KV cache              0.22 GB  (10.0 KB/token, hybrid: 10/40 full-attention layers)
  prefill workspace     0.09 GB  (estimate, at --prefill-step-size 128)
  runtime reserve       1.00 GB  (buffer cache, activations, fragmentation)
                       ----------------------------------
  peak                 13.89 GB of 14.40 GB usable   0.51 GB headroom

Verdict: ⚠️  RESIDENT — fits, but only after raising the Metal wired cap

Recommended:
  sudo sysctl -w iogpu.wired_limit_mb=13721
  --prefill-step-size 128
  --kv-bits 8

Planning for a machine you're not sitting at — say, deciding whether to buy or use a mini — is what --wired-gb / --ram-gb are for (the run above is exactly that: a 16 GB mini, projected from a 64 GB machine). turboquant-doctor adds a readiness check, and both take --json.

The projection is calibrated against real measurements on the mini rather than estimated from theory: it predicts a 10.44 GB peak where the 9.4 GB ternary sibling measures 10.42, and the wired limit it suggests here lands within 1% of the 13824 that this build actually runs on.

As an agent backend (OpenAI-compatible server)

pip install "turboquant-mlx-full[serve]>=0.13"

turboquant-serve \
    --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
    --kv-bits 8 --tool-syntax-greedy

--tool-syntax-greedy forces argmax inside <tool_call> blocks (keys, punctuation, tags) while leaving payload strings and the call-or-not decision sampled — cheap insurance for tool-call syntax at low bit-widths. Use temperature ~0.7; greedy-everything hurts this model's planning.

Quality — measured, not assumed

Agentic harness test (Opencode, fix-the-failing-test task, identical server/harness/prompt across builds, 64 GB M4 Max):

Build	Experts	Size	Opencode result
pure ternary (tq3a-tqTe-g64)	1.58-bit ×3	9.4 GB	❌ 0/4 — command loops, hallucinated tool args
ternary + 3-bit down_proj	down3	11.6 GB	❌ 0/3
ternary + 4-bit down_proj (this)	down4	12.6 GB	✅ 3/3 — full test → diagnose → fix → green loop
3-bit everything (tq3-g32)	3-bit	16.4 GB	✅ (control, 33 s)

Measured on a 16 GB Mac mini

Validated end-to-end on a base M-series 16 GB Mac mini (model on an external SSD), serving over the OpenAI-compatible API:

Metric	Result
Decode	15.2–15.6 tok/s, flat from 300 to 1800 generated tokens
Peak memory (generate)	13.6 GB
Cold prefill	21K-token document in 234 s (90 tok/s, 256-token chunks)
Opencode agentic task	✅ 3/3 (fix-the-failing-test: 3:37 / 2:25 / 1:54 — faster each run as the disk cache warms)
Follow-up-turn cache reuse	20,480 / 21,250 tokens restored from disk checkpoints (96.4%)
Long-form generation	coherent with the shipped repetition_penalty: 1.05 (this repo's generation_config.json)

The validated serving recipe (requires turboquant-mlx-full ≥ 0.13):

sudo sysctl -w iogpu.wired_limit_mb=14336   # 14 GiB; 13824 works below ~16K context

turboquant-serve \
    --model manjunathshiva/Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
    --host 0.0.0.0 --port 8080 \
    --kv-bits 8 --tool-syntax-greedy --disk-cache \
    --prefill-step-size 128 \
    --temp 0.7 --top-p 0.8 --top-k 20 \
    --chat-template-args '{"enable_thinking": false}' \
    --prompt-concurrency 1

Point Opencode (or any OpenAI-compatible agent harness) at http://<host>:8080/v1 with model id default_model, temperature ~0.7, and auto-discovered skills disabled. Practical context ceiling on 16 GB is ~16–18K tokens at the default wired cap; the --prefill-step-size 128 + 14 GiB wired combination above is what carries 21K+ (the serve defaults --metal-cache-limit-gb auto and --prompt-cache-max-gb auto handle the rest of the tight-memory story automatically).

Choosing a build

Build	Size	Best for
tq3a-tqTe-g64	9.4 GB	chat, drafting, Q&A on a 16 GB mini — default wired cap, no sudo
tq3a-tqTe-down4-g64 (this)	12.6 GB	agent harnesses / tool loops on a 16 GB mini (needs the sysctl bump)
tq3-g32	16.4 GB	recall-critical work, exact literals; 24 GB+ machines

How it was made

python -m turboquant_mlx.convert \
    --hf-path Qwen/Qwen3.6-35B-A3B \
    --mlx-path Qwen3.6-35B-A3B-tq3a-tqTe-down4-g64 \
    --bits 3 --group-size 64 --ternary-experts --expert-down-bits 4

Data-free (no calibration set): randomized Hadamard rotation → per-group RMS scaling → Lloyd-Max codebook (ternary for up/gate, 16-level for down) → trit/bit packing. Per-projection bit-width is self-describing on disk via codebook length; expert_down_bits is recorded in config.json so converter and loader agree.

Links

• TurboQuant-MLX on GitHub (Apache-2.0)
• PyPI: turboquant-mlx-full
• Siblings: pure ternary 9.4 GB · 3-bit 16.4 GB · Qwen3.5-122B ternary