YanissAmz/MiMo-V2.6-Flash-RL-GGUF

MiMo-V2.6-Flash-RL — IQ2_M-class GGUF (100 GiB), fits 128 GB machines

A single-quant GGUF of XiaomiMiMo/MiMo-V2.6-Flash-RL (309B total / 15B active, 256 experts, top-8), 2.76 bits per weight on average, sized for machines with ~110 GB of free RAM/VRAM: 128 GB unified-memory systems (AMD Ryzen AI Max+ 395 / Strix Halo, Apple Silicon 128 GB, DGX Spark), multi-GPU rigs, or CPU + GPU offload. It is a standard GGUF: any llama.cpp build and backend (CUDA, ROCm/HIP, Vulkan, Metal, CPU) runs it.

Size	100.4 GiB in 3 shards (99.5 GiB model + 1 GiB MTP layers), 2.76 bpw average
Quant type	custom IQ2_M-class mix (see Recipe); the header reports IQ2_M
KLD vs native MXFP4	0.164 mean · 0.021 median · 2.31 p99
Same top-1 token as MXFP4	88.2 %
PPL ratio vs MXFP4	1.128
Decode, example: AMD Strix Halo (Vulkan)	26.1 tok/s (tg128)
Prefill, example: AMD Strix Halo (Vulkan)	~310 tok/s at -ub 2048 (pp4096)

Loads and generates on stock llama.cpp (tested: master of 2026-09-23, CPU build). The speeds above are one example machine (AMD Ryzen AI Max+ 395, Radeon 8060S, llama.cpp Vulkan build); other hardware will differ.

Files

MiMo-V2.6-Flash-RL-IQ2_M-00001-of-00003.gguf
MiMo-V2.6-Flash-RL-IQ2_M-00002-of-00003.gguf
MiMo-V2.6-Flash-RL-IQ2_M-00003-of-00003.gguf

Point llama.cpp at the first shard.

Usage

llama-server -m MiMo-V2.6-Flash-RL-IQ2_M-00001-of-00003.gguf \
  -ngl 99 -fa on -c 32768 -b 2048 -ub 2048 \
  --cache-type-k q8_0 --cache-type-v q8_0 --jinja

-ub 2048 matters for MoE prefill (Strix Halo example: 193 tok/s at the default -ub 512, 310 tok/s at 2048).

Recipe

The experts of MiMo-V2.6-Flash are stored natively in MXFP4, so the ggml-org MXFP4 GGUF is a lossless reference for them. Every number above is measured against it.

• Per-tensor bit allocation under a 99.5 GiB budget, minimizing the imatrix-weighted weight error of each expert tensor multiplied by the residual-stream share of its layer. The per-tensor types are chosen for quality per byte, not for one hardware target. MiMo has massive-activation layers (layer 7 carries ~96 % of the residual norm, layers 5, 2, 6, 47 and 1 also high). Quantizing those to 2 bits breaks the model (KLD 3.4, PPL ×27 in a uniform-allocation control), so they keep MXFP4 on most of their experts.
• Experts: IQ2_XXS / IQ2_XS / IQ2_S / IQ3_XXS / IQ3_S / Q3_K by tensor, MXFP4 on the high-share layers.
• Attention QKV, attention output and the output head: Q6_K (Q8_0 → Q6_K on attention output and head costs +0.4 % KLD, within noise, and gives +6.7 % decode). Embeddings and the dense first layer: Q8_0.
• Importance matrix: AesSedai's imatrix for this model.

Evaluation

• Reference: native MXFP4 GGUF logits (CPU run).
• Corpus: 115 KB of mixed text (Wikipedia, code, calibration-style text), 13 chunks × 2048 tokens, llama-perplexity --kl-divergence.
• Speed (example machine): llama-bench, Vulkan (RADV), Radeon 8060S, idle machine, -fa 1.

MTP layers

The file also contains the model's 3 multi-token-prediction layers (Q8_0, ~1 GiB). Stock llama.cpp ignores them ("unused tensor" warnings are expected). In our measurements, MTP self-speculation does not speed up decode yet on our test hardware (draft acceptance ~50 %); keep speculation off for now.

License

MIT, same as the base model.