--- license: mit base_model: XiaomiMiMo/MiMo-V2.6-Flash-RL base_model_relation: quantized pipeline_tag: text-generation library_name: transformers tags: - vllm - modelopt - nvfp4 - quantized - moe - mimo - mimo-v2.6 - agentic - tool-calling - blackwell ---

Xiaomi's own 4-bit experts, re-encoded for Blackwell's FP4 tensor cores.

NVFP4 build of XiaomiMiMo/MiMo-V2.6-Flash-RL at 182.40 GB, on two 96 GB cards.
Every routed-expert weight keeps its released value; qkv stays FP8 and everything else BF16, as released.

> **There is no BF16 to start from.** Xiaomi ships this model's routed experts as MXFP4 and its attention > as FP8, so the release is already a 4-bit checkpoint and it is the reference row in every table > below. This build changes the expert format, not the expert values. --- ## Why this quant * 🔁 **The release's weights, bit for bit.** All 302,795,194,368 routed-expert values reproduce the released MXFP4 values exactly. Each E8M0 scale over 32 weights becomes two E4M3 scales over 16 against a power-of-two global, and no value needed rounding. * ⚡ **The experts run on native FP4.** vLLM serves them with FlashInfer CUTLASS W4A4 on sm_120. Over the full knowledge suite at concurrency 16 that is 532 tok/s against the release's 402. * 🎯 **Tool-calling level with the release.** 78.6 over five runs, the release also 78.6. Knowledge 91.1 against 90.3. * 📏 **Activation scales set per layer.** FP4 activations need one global scale per layer. The last MoE layer's expert intermediate reaches about 60,000 on the prompts we measured, 22× past what a scale of 1.0 can hold, so a single constant would clip it. * 🧮 **The cost is KV cache.** NVFP4 stores 4.5 bits per weight to MXFP4's 4.25, so this build is 9.48 GB larger than the release. On two 96 GB cards that leaves a 250,094-token pool, against 627,110 for the release. --- ## Serve it ```bash hf download primitive-ai/MiMo-V2.6-Flash-RL-NVFP4 --local-dir ./MiMo-V2.6-Flash-RL-NVFP4 SP=/usr/local/lib/python3.12/dist-packages/vllm docker run --gpus all --ipc=host --shm-size 32g -p 8000:8000 -v $PWD:/models \ -v $PWD/MiMo-V2.6-Flash-RL-NVFP4/vllm_patch/mimo_v2.py:$SP/model_executor/models/mimo_v2.py:ro \ vllm/vllm-openai:mimo-v26-x86_64-cu130 \ --model /models/MiMo-V2.6-Flash-RL-NVFP4 \ --tensor-parallel-size 2 --trust-remote-code --generation-config vllm \ --max-model-len 131072 --gpu-memory-utilization 0.95 \ --limit-mm-per-prompt '{"image":4,"video":0,"audio":0}' \ --reasoning-parser mimo --tool-call-parser mimo --enable-auto-tool-choice ``` The day-0 image (vLLM `0.29.1rc1.dev449`) loads ModelOpt mixed-precision checkpoints, but its MiMo loader assumes the release's own FP8 layout: the fused qkv loader writes to `weight_scale_inv`, while ModelOpt's block-FP8 layers name that parameter `weight_scale` and store it 4-D. `vllm_patch/mimo_v2.py` is the image's file with those spots bridged; the 21-line diff is `vllm_patch/mimo_v2.diff`. Without the mount, loading stops with a `KeyError`. Setting audio to 0 skips loading the audio encoder. The DFlash drafter ships as released, but speculative decoding was not measured with this build. --- ## Measured Two RTX PRO 6000 Blackwell, 96 GB each, tensor parallel 2. The 1,170-item knowledge suite and the 200-item tool-calling suite, `temperature 0.6` / `top_p 0.95` / `top_k 20`, thinking on, a 16,384-token budget, concurrency 16, auto-scored with no LLM judge. bf16 KV, no speculative decoding, same box, same day. | build | size | knowledge | tool-calling | call | abstain | finished | tok/s @1 | tok/s @16 | |---|---|---|---|---|---|---|---|---| | [release](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL), MXFP4 experts | **172.92 GB** | 90.3 | 78.6 | 82.8 | 62.0 | 99.0% | 112.2 | 402 | | **this repo, NVFP4** | 182.40 GB | **91.1** | 78.6 | 83.0 | 61.0 | 99.0% | **121.1** | **532** | Tool-calling is the mean of five runs per build, with a within-build standard deviation of 0.8 to 0.9, so the column is one band. Knowledge is one run per quantized build and two for the release; gaps under 1.2 are ties. `tok/s @1` is 60 real prompts one at a time; `tok/s @16` is the knowledge suite's own aggregate. --- ## Comparable with our other models Accuracy numbers move for reasons that have nothing to do with the model: a shorter token budget, a different `temperature`, or whether the model was allowed to reason at all. So every number in this table, on this card and on our other cards, comes from the one fixed protocol described above, the same 1,370 items, auto-scored, no LLM judge. | model | shape | size | overall | knowledge | call | abstain | finished | out/answer | |---|---|---|---|---|---|---|---|---| | [Laguna-XS-2.1](https://huggingface.co/primitive-ai/Laguna-XS-2.1-mixed-NVFP4-MXFP8) | 31 B MoE | 19.3 GiB | 81.7 | 83.8 | 68.4 | 73.5 | 98.9% | 1097 | | [Nemotron-3.5-Lightning-30B-A3B](https://huggingface.co/primitive-ai/Nemotron-3.5-Lightning-30B-A3B-mixed-INT4-INT8) | 30 B MoE+Mamba | 19.2 GiB | 87.1 | 87.9 | 85.4 | 70.5 | 97.9% | 1429 | | [Ornith-1.5-35B-A3B](https://huggingface.co/primitive-ai/Ornith-1.5-35B-A3B-mixed-NVFP4-FP8) | 35 B MoE | 22.6 GiB | 88.7 | 91.7 | 74.4 | 60.0 | 99.3% | 760 | | [Muse-Glimmer-30B](https://huggingface.co/primitive-ai/Muse-Glimmer-30B-mixed-NVFP4) | 30 B MoE | 20.4 GiB | 86.6 | 88.8 | 78.6 | 54.5 | 99.7% | 800 | | [Qwen3.8-27B](https://huggingface.co/primitive-ai/Qwen3.8-27B-mixed-NVFP4-FP8) | 27 B dense | 20.7 GiB | 88.8 | 90.4 | 85.5 | 54.5 | 99.7% | 651 | | [Granite-4.2-30B](https://huggingface.co/primitive-ai/granite-4.2-30b-mixed-NVFP4-FP8) | 30 B dense | 18.1 GB | 85.5 | 86.2 | 85.8 | 60.8 | 98.5% | 1502 | | [Nex-N2.5-mini NVFP4](https://huggingface.co/primitive-ai/Nex-N2.5-mini-NVFP4) | 35 B MoE, 3 B active | 23.91 GB | 88.5 | 90.5 | 81.9 | 57.5 | 99.3% | 524 | | [Nex-N2.5-mini mixed](https://huggingface.co/primitive-ai/Nex-N2.5-mini-mixed-NVFP4-FP8) | 35 B MoE, 3 B active | 26.04 GB | 88.5 | 90.6 | 80.9 | 58.7 | 99.2% | 504 | | [Nex-N2.5-mini FP8](https://huggingface.co/primitive-ai/Nex-N2.5-mini-FP8) | 35 B MoE, 3 B active | 38.13 GB | 88.7 | 90.9 | 79.4 | 60.0 | 99.5% | 545 | | [K2-Horizon-MoVA-36B-A4B NVFP4](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-NVFP4) | 37 B MoE+MoVA, 4 B active | 36.7 GB | 84.1 | 86.5 | 71.8 | 60.4 | 95.6% | 1234 | | [K2-Horizon-MoVA-36B-A4B mixed](https://huggingface.co/primitive-ai/K2-Horizon-MoVA-36B-A4B-mixed-NVFP4-FP8) | 37 B MoE+MoVA, 4 B active | 44.5 GB | 84.9 | 87.3 | 73.5 | 58.9 | 96.3% | 1118 | | [Laguna-S-2.1](https://huggingface.co/primitive-ai/Laguna-S-2.1-mixed-NVFP4-MXFP8) | 110 B MoE | 64.0 GiB | 84.3 | 87.1 | 64.6 | 81.0 | 97.3% | 995 | | [Qwen3.8-Flash-Next](https://huggingface.co/primitive-ai/Qwen3.8-Flash-Next-mixed-NVFP4-FP8) | 180 B MoE, 6 B active | 183.7 GB | 90.3 | 92.2 | 84.8 | 56.7 | 99.5% | 686 | | **MiMo-V2.6-Flash-RL NVFP4** (this repo) | 309 B MoE, 15 B active | **182.40 GB** | **89.3** | 91.1 | 83.0 | 61.0 | 99.0% | 664 | `overall` pools the two suites as 1,370 items, weighted 85.4% knowledge and 14.6% tool calling by item count. **Read it with `finished`**: `overall` scores an answer that overran the token budget as wrong, and cannot say whether the model needed the room or failed to stop. A gap under **1.0** in `overall` is a tie. Sizes are as each card reports them, which mixes GB and GiB. --- ## What's quantized to what MiMo-V2.6-Flash-RL is 309 B parameters, 15 B active, and 98% of those parameters are routed experts. | tensors | count | format | |---|---|---| | routed experts, layers 1 to 47 | 36,096 projections | NVFP4: E2M1 codes, E4M3 scale per 16, FP32 global, W4A4 | | `qkv_proj` ×48, layer-0 dense MLP, MTP layers | 63 modules | FP8 E4M3, 128×128 blocks, byte-identical to the release | | `o_proj`, embeddings, `lm_head`, routers, vision and audio encoders | the rest | BF16, byte-identical to the release | ModelOpt `MIXED_PRECISION`, one `quantized_layers` entry per module; the dense FP8 entries are tagged `FP8_PB_WO`, the name vLLM's ModelOpt linear table accepts for block FP8. gate and up of each expert share one global scale, so the fused projection vLLM builds keeps the exact values. vLLM reduces expert input scales to one per layer, and the checkpoint stores them that way. The expert weights are not re-quantized. The activation scales are the only new numbers in this checkpoint. ---

_{primitive ·
more models ·
inference economics for production LLM systems}