Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-mlx

Qwen3.8-Flash-Next pruned to 99B total / ~5.4B active parameters, stored at 3-bit — 4.1× less expert memory than the 4-bit reference, 36.7 GB on disk: half the bytes of the smallest published quant, and it fits a 48 GB Mac. Stock mlx-vlm plus the small loader shipped in this repo.

Niwaki (庭木) are Japan's garden trees, sculpted by meticulous pruning so that every branch serves the form of the whole. This model applies that spirit to a Mixture-of-Experts: routed experts are pruned using statistics of their routed tokens over a mixed web/code/chat/reasoning corpus, reconstructed to compensate, then distilled from the full model on the same mixed corpus, and stored at low precision. The result is the same architecture with fewer, narrower experts.

A paper with the full method and measurements is coming soon.

Benchmarks

Evaluation protocol: WikiText-2 perplexity over 64 × 2048-token windows (fp32 loss, no chat template); task average over arc_easy, hellaswag, piqa, winogrande and boolq (zero-shot, 500 examples each), paired against the reference on the identical examples.

model	expert storage	wt2 ppl ↓	task avg ↑	retention	generation d2 (avg/min)
reference (4-bit)	75.5 GB	5.06	0.759	100%	0.78 / 0.62
this model	18.5 GB (0.245×)	7.88	0.733	96.6%	0.68 / 0.51

Generation quality is measured, not assumed: an 8-prompt battery (code, reasoning, chat, creative; 600-token sampled generations) scored by bigram diversity. Known limitation (measured): on multi-step arithmetic prompts the model reasons longer and more repetitively than the reference (battery minimum 0.51 on those prompts, and it uses the whole 600-token budget where the reference stops early); chat, explanations and short-form writing are on par (0.90–0.93).

Model dimensions

total parameters	99B (from ~177B)
active parameters per token	~5.4B (from ~6.7B)
layers / routed experts / top-k	48 / 512 / 10 (routed experts kept on 24 of the 48 layers)
expert intermediate size	448 (from 640)
expert precision	3-bit (group 64)
n-gram embedding table (PLE) precision	2-bit (group 128)
backbone precision	4-bit (untouched)
whole-model size on disk	36.7 GB
runs comfortably in	48 GB unified memory
modality	text only (vision tower not included)

Usage (MLX, Apple Silicon)

Requires mlx-vlm >= 0.7.0rc0 and mlx >= 0.32, plus niwaki_flash_load.py from this repo (it tells stock mlx-vlm about the artifact's layout).

pip install -U mlx mlx-vlm transformers
hf download neopolita/Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-mlx --local-dir ./niwaki-flash

import mlx_vlm
from niwaki_flash_load import load          # niwaki_flash_load.py from this repo

model, processor = load("./niwaki-flash")
messages = [{"role": "user", "content": "Explain hyper-connections in two sentences."}]
text = processor.tokenizer.apply_chat_template(messages, add_generation_prompt=True,
                                               tokenize=False, enable_thinking=False)
print(mlx_vlm.generate(model, processor, text, max_tokens=300, verbose=False).text)

enable_thinking=False skips the reasoning preamble; leave it on for harder prompts. On an M4 Max: ~30 tokens/s, ~37 GB peak.

GGUF builds for llama.cpp (CUDA, Vulkan, Metal, CPU; they need the neopolita-llama.cpp fork): Qwen3.8-Flash-Next-99B-A5B-Niwaki-3bit-gguf (UD-Q3K 42.5 GB recommended, UD-Q4K 54.1 GB).

Notes and limitations

• Compression trades quality: this model sits at ~97% of the reference task average and 1.56× its perplexity. Fluent and on-topic; numbers and niche technical facts are less reliable than the reference.
• Evaluated text-only on English-web-heavy data; the base model's biases are inherited and rare-domain behaviour of pruned experts is less tested.
• The Niwaki family on Qwen3.6-35B-A3B: 27B-A3B-Niwaki-2bit · 19B-A3B-Niwaki-2bit · 11B-A3B-Niwaki-4bit.

Base model by the Qwen team (see license); 4-bit MLX conversion by mlx-community; pruning and distillation by the Niwaki project, 2026-09. Method details: paper coming soon.