Qwen3-235B-A22B-Instruct-2507-tq3a-tqTe-g64

Ternary (1.58-bit) expert TurboQuant quantization of Qwen/Qwen3-235B-A22B-Instruct-2507 — 3-bit attention, ternary {-c, 0, +c} experts packed as base-3 trits (group size 64) — produced with TurboQuant-MLX.

A 235B-parameter MoE compressed from 470 GB (BF16) to 53.2 GB — small enough to run fully resident on a 64 GB Apple Silicon Mac at **5.6 tok/s**, with no expert streaming.

Model Details

• Base model: Qwen/Qwen3-235B-A22B-Instruct-2507 — Mixture-of-Experts (qwen3_moe)
• Architecture: 94 layers, hidden size 4096, 128 routed experts (top-8), MoE intermediate 1536, ~235B total / ~22B active params
• Quantization: TurboQuant (Hadamard rotation + Lloyd-Max codebook), group size 64
  • Attention → 3-bit (q/k/v/o_proj + lm_head)
  • Experts → ternary 1.58-bit (gate/up/down_proj of every expert): the data-free {-c, 0, +c} codebook (optimal Lloyd-Max for N(0,1), c ≈ 1.224), with the {0, 1, 2} indices packed as genuine base-3 trits — 20 per uint32 (3²⁰ < 2³²) = ~1.6 bpw, vs 2.0 for a bit-packed 2-bit slot
  • Routers → full precision (mlp.gate is auto-skipped — never quantized)
• Size: 53.18 GB across 11 shards (vs ~470 GB BF16 — an 8.8× reduction)

Why ternary (tq3a-tqTe)?

The experts dominate the parameter count, so the memory story is set by how few bits they take. Adding the zero level to a two-sided codebook ({-c, 0, +c} vs {-c, +c}) is what makes sub-2-bit work data-free: a plain 1-bit expert tier collapses into word-salad, while ternary lands at ~2-bit reconstruction quality (normalized MSE ≈ 0.19). Packing those three levels in base 3 (not the 2-bit slot) is a real 1.6 bits/weight, which is what pulls this build down to 53 GB and inside a 64 GB machine's resident budget.

Ternary needs expert redundancy to average out its coarser quantization: Qwen3-235B routes top-8 of 128 experts per token, and that redundancy is more than enough — the same effect that lets 2-bit experts hold on 128-expert MoEs while 32-expert models fall apart. The shared attention path is hit on every token with no expert averaging to hide errors, so it stays at the safer 3-bit. Routers stay full precision because a wrong expert selection is unrecoverable.

Note: this is an instruct model. Use the chat template (the generator applies it automatically).

Will it fit my Mac?

Don't guess, and don't download 57.1 GB to find out — ask first (needs turboquant-mlx-full >= 0.15.1):

turboquant-plan --model manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tqTe-g64

It reads only this repo's safetensors headers over the network (a couple of hundred KB, a few seconds — the weights are never fetched), then projects the peak against your machine and prints the flags to use:

Model
  weights (exact)      57.1 GB
    experts            52.6 GB  (streamable)
    resident           4.5 GB
  KV cache             188.0 KB/token  (all layers full-attention)

Verdict: <resident | needs a wired-limit raise | streaming>
Recommended: <the exact flags for your Mac>

Planning for a machine you're not sitting at — deciding whether a 16 GB mini or a 64 GB Mac can take this — is what --wired-gb / --ram-gb are for:

turboquant-plan --model manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tqTe-g64 \
    --wired-gb 10.5 --ram-gb 16 --context 8192

turboquant-doctor runs the same projection plus a readiness check; both take --json. The projection is calibrated against real measurements on a 16 GB Mac mini rather than estimated from theory — on the 9.4 GB ternary 35B it predicts a 10.44 GB peak where that machine measures 10.42.

Quality

Validated with a 6-probe stress suite on a 64 GB Apple Silicon Mac (fully resident, greedy decode, temp = 0). All six probes pass:

Probe	Result
In-context needle recall	✅ Returned the access code exactly (7-premium-4823-November)
Multi-step math	✅ 23 × 17 = 391
Code generation	✅ Correct is_prime with the √n odd-divisor optimization
Strict JSON formatting	✅ Emitted the requested object ({"name": "Alice", "age": 30, "city": "Paris"})
Reasoning	✅ Textbook Rayleigh-scattering explanation of why the sky is blue
List / ordering	✅ All eight planets in order (Mercury→Neptune) with a Pluto footnote

A free-form 160-token generation on photosynthesis was fully coherent and correct (light-dependent vs Calvin-cycle stages, photosystems II/I, thylakoid membrane, ATP/NADPH).

Notably, exact literal recall was correct in testing — the leading-digit substitution that can affect 2-bit-expert MoEs (see the tq3a-tq2e sibling) did not appear here at this group-size-64 ternary configuration. As always for lowest-bit tiers, if you need guaranteed exact recall of long codes/IDs, prefer the full-3-bit sibling below.

Choosing a build

Build	Experts	Size	Fits resident	Notes
tq3a-tqTe-g64 (this)	ternary ~1.6 bpw	53 GB	64 GB	smallest; fully resident on a 64 GB Mac
tq3a-tq2e-g32	2-bit	70.5 GB	96 GB+ (streams on 16/64 GB)	hybrid; slight leading-digit recall flip
tq3-g32	3-bit	103 GB	96 GB+	recall-critical; exact literals

Running it (fully resident on a 64 GB Mac)

Requires turboquant-mlx-full ≥ 0.12.3 (the base-3 trit decode kernels; ≥ 0.12.1 is mandatory for expert streaming of this build — earlier versions crash on the trit format — and 0.12.3 auto-applies this repo's sampling defaults).

pip install "turboquant-mlx-full>=0.12.3" mlx-lm

# Raise the Metal wired-memory cap above the ~48 GB default so a 53 GB model
# stays fully GPU-resident (one-time, per boot):
sudo sysctl -w iogpu.wired_limit_mb=60416   # 59 GiB; leaves ~5 GB for the OS

python -m turboquant_mlx.generate \
    --model manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tqTe-g64 \
    --prompt "Explain why the sky is blue." \
    --max-tokens 512

Measured on a 64 GB M4 Max (fully resident, greedy):

Metric	Value
Load	~10 s
Decode	~5.6 tok/s
Prefill	~5.8 tok/s
Peak memory	58.4 GB

The remaining decode-speed ceiling is compute-bound (the online Hadamard rotation), not memory — raising the wired cap further removes disk paging but does not lift throughput much beyond this.

Under 64 GB? Fall back to expert streaming (pages only the router-selected experts from disk):

python -m turboquant_mlx.stream.stream_generate \
    --model manjunathshiva/Qwen3-235B-A22B-Instruct-2507-tq3a-tqTe-g64 \
    --prompt "Explain why the sky is blue." \
    --max-tokens 512 --cache-budget-gb 6

How TurboQuant works

TurboQuant is data-free — no calibration set. Each weight matrix is rotated by a randomized Hadamard transform (Gaussianizing the coordinate distribution), grouped, RMS-scaled per group, and mapped to a Lloyd-Max codebook that is optimal for N(0,1); ternary experts use the 3-level {-c, 0, +c} codebook and store their indices as base-3 trits. Inference applies the (orthonormal) rotation to the activations and decodes the trits inline in a fused Metal kernel, so the packed weight never leaves its ~1.6-bpw form in memory.

License

Apache-2.0, inherited from the base model Qwen/Qwen3-235B-A22B-Instruct-2507. Quantized by Manjunath Janardhan with TurboQuant-MLX.