TrevorJS/MiMo-V2.6-Flash-RL-GGUF

MiMo-V2.6-Flash-RL GGUF for a single DGX Spark

An imatrix-calibrated IQ2 build of XiaomiMiMo/MiMo-V2.6-Flash-RL (309B total / 15B active MoE, 1M context) that fits and runs on one DGX Spark (GB10, 128 GB unified memory), plus a GGUF of Xiaomi's DFlash drafter. Converted from snapshot 3b38d063.

This is v2.1. It replaces the first upload (v1), which was 22% slower at decode and scored lower on GSM8K; see Changes from v1.

Files

file	size	what
MiMo-V2.6-Flash-RL-IQ2_XS_M-v2.1-*.gguf	91.9 GiB, 10 shards	the model; load the first shard
MiMo-V2.6-Flash-RL-DFlash-Q8_0.gguf	2.1 GiB	DFlash speculative drafter (see DFlash)
recipe-iq2_xs_m-v2.1.txt		--tensor-type-file used with llama-quantize

Text only. The vision and audio encoders convert separately (mmproj) and are not included.

Measured on a DGX Spark (GB10)

Speed

llama-bench, -fa 1 -ngl 99:

test	-ub 512 (default)	-ub 2048	-ub 4096
pp2048	421.7	733.5	730.0
pp8192	418.6	725.1	799.5
pp2048 at 8K depth		719.3
tg32	32.1

Pass -ub 2048. The default micro-batch of 512 leaves ~45% of prefill on the table: at 512 each of the 256 experts sees too few tokens per matmul.

Server, 512-token greedy generations, thinking off:

config	code	reasoning	prose
no draft	30.6	30.7	30.8
DFlash, 5 drafted, p_min 0.7	42.9	38.7	26.9

DFlash helps code (+40%) and reasoning (+26%) and costs prose about 13%, where the drafter is rarely right. For prose-heavy use, run without it.

A 5.5K-token prompt at -ub 2048: prefill 671 t/s without the drafter, 656 t/s with it at -b 8192 (647 at -b 2048); decode on the reply 29 vs about 33 t/s. The drafter only reads the prompt's last 1024 positions, so only those are fed to it; a larger -b means fewer feeds per prompt.

Quality

GSM8K, 500 items, greedy, zero-shot, thinking off. The items and prompt length are matched to the harness used for benthecarman/MiMo-V2.6-Flash-RL-exl3 (the same 500 test items, prompts totalling the same 49,205 tokens), so the columns are comparable:

build	GSM8K
FP8 reference, unquantized (from the EXL3 card)	96.4%
EXL3 2.36 bpw (from the EXL3 card)	95.8%
this build (v2.1)	95.6% (478/500)
v1 of this repo	95.0%

Perplexity on wiki.test.raw (full file, -c 512): 5.7225 ± 0.034, which is +11.2% over the BF16 reference of 5.1455 reported by AesSedai with the same method. Arithmetic survives this bitrate far better than perplexity suggests.

Recipe

tensors	type
routed expert gate, up	IQ2_XS
routed expert down	IQ2_S
layer 47 routed experts (gate, up, down)	Q6_K
attention QKV and output	IQ4_XS
dense FFN (layer 0, MTP blocks), MTP eh_proj	Q6_K
token embeddings	Q8_0
output head	Q6_K
router, norms, sinks, expert bias	F32

2.55 bits per weight overall.

Why attention is IQ4_XS: on this model the non-expert tensors are over half of what each token reads from memory at IQ2, so dropping attention from Q6_K to IQ4_XS bought 22% decode speed with no measurable GSM8K cost.

Why layer 47 is Q6_K: layer 47's expert intermediates (the input to its down projection) get large enough that a 32-value sum passes f16's 65504. In llama.cpp's CUDA kernels, Q4_K and Q5_K (and Q4_0/Q4_1/Q5_0/Q5_1/Q2_K/IQ1_S) store the activation block sum in f16, so a build with layer 47 at Q5_K turned every perplexity chunk into NaN. Q6_K and the IQ types use an f32 scale and are unaffected, so this file runs correctly on stock llama.cpp.

llama-quantize --allow-requantize \
  --imatrix <imatrix.gguf> \
  --tensor-type-file recipe-iq2_xs_m-v2.1.txt \
  --token-embedding-type q8_0 --output-tensor-type q6_k \
  MiMo-V2.6-Flash-MXFP4_MOE.gguf out.gguf IQ2_XS

--allow-requantize is required because the source experts are already 4-bit.

Calibration

The importance matrix is Baekpica's, collected from the full MXFP4 model on text and multimodal inputs. It covers 12,030 of 12,032 layer/expert slots; the two never observed are block 7 experts 13 and 184, which our own calibration runs never reached either, so they look unrouted rather than under-sampled.

Running it

Needs llama.cpp with mimo2 support, including #28865 (older loaders reject the per-layer sliding-window pattern).

llama-server -m MiMo-V2.6-Flash-RL-IQ2_XS_M-v2.1-00001-of-00010.gguf \
  -ngl 99 -fa on -c 16384 -b 2048 -ub 2048 --jinja

Memory: about 92 GiB of weights leaves ~25 GiB on a 128 GB Spark. Contexts of 16K were fine throughout our testing. A llama-bench run with a 32K prompt froze our machine (unified memory does not OOM-kill; it stalls), so measure headroom before going much larger.

KV is cheap regardless of context: only 9 of the 48 layers are global attention, 22.5 KB per token at F16 (2.7 GiB at 128K); the other 39 keep a 128-token sliding window.

DFlash drafter

MiMo-V2.6-Flash-RL-DFlash-Q8_0.gguf is Xiaomi's 5-layer block-diffusion drafter from the release's dflash/ directory, at Q8_0. Stock llama.cpp does not run DFlash for MiMo yet. Anyone porting it should know three things the bundled Python example leaves out, all of which this file already encodes:

• The mask token needs its learned vector. The target's embedding row for the mask token (151675) is all zeros, never trained. The drafter was trained with the vector in mask_embedding.pt. This GGUF carries its own token_embd with that row replaced; a runtime that embeds mask slots through the target's table feeds the drafter zeros.
• Partial rotary: 64 of 128 head dimensions rotate (partial_rotary_factor 0.5); rope.dimension_count is set.
• Value scale 0.612 on the attention output (attention.value_scale).

Target features come from layers 0/11/23/35/47 (the residual stream before the final norm), 8-token blocks, non-causal within the block, 1024-token sliding window. --spec-draft-p-min 0.7 (cut a block at the first low-confidence token) was the best setting we measured; without it (on v1, 7 drafted) prose ran at 0.57x.

The three MTP blocks are included in the main file but not recommended. MiMo's MTP heads are not chained: head k reads the target's hidden state, with the token k places further ahead (SGLang pairs them this way). llama.cpp's multi-head driver feeds each head the previous head's output instead, which makes heads 2 and 3 draft noise. With the pairing fixed, drafting one token gives 0.97x overall and deeper drafts are slower on this MoE, so DFlash remains the faster option.

Changes from v1

	v1	v2.1
size	90.1 GiB	91.9 GiB
decode (tg32)	26.3 t/s	32.1 t/s
GSM8K	95.0%	95.6%
wiki PPL	5.7824	5.7225
attention	Q6_K	IQ4_XS
layer 47 experts	IQ2_XS / IQ2_S	Q6_K
imatrix	ours, from a Q2_K bootstrap	Baekpica's, from the full model

Architecture notes

48 layers, 39 sliding-window (window 128) and 9 global, interleaved. 64 query heads at head_dim 192 (QK) / 128 (V); 4 KV heads on global layers, 8 on SWA. 256 routed experts, 8 active, no shared expert, sigmoid routing. Layer 0 is dense.

The release stores the routed experts as MXFP4 (one E8M0 exponent per 32 values) under quant_method: "fp8" with store_dtype: "mxfp4"; QKV is block-scaled FP8 sharded across TP=4. llama.cpp converts it natively as of #29257.