README.md

---
license: mit
base_model:
  - XiaomiMiMo/MiMo-V2.6-Flash
  - patrickbdevaney/MiMo-V2.6-Flash-REAP50
tags:
  - gguf
  - llama.cpp
  - reap
  - hope
  - moe
  - pruned
  - multimodal
  - vision
  - audio
  - mtp
---

# Xiaomi MiMo-V2.6-Flash REAP-50 — GGUF

Official GGUF quantisations of **MiMo-V2.6-Flash-REAP50**, a 50% routed-expert pruned checkpoint of `XiaomiMiMo/MiMo-V2.6-Flash` created with [REAP](https://github.com/CerebrasResearch/reap) and **HOPE** second-order saliency pruning.

* **Base HF Checkpoint**: [patrickbdevaney/MiMo-V2.6-Flash-REAP50](https://huggingface.co/patrickbdevaney/MiMo-V2.6-Flash-REAP50)
* **Experts Retained**: **128 of 256** routed experts per layer across 47 MoE layers (1 dense layer, 47 MoE layers).
* **Base Architecture**: Native packed **MXFP4** (`U8`, block size 32) experts with unquantized pure **BF16** attention and embeddings.
* **Towers Included**: Vision & Audio multimodal projectors (`mmproj`) and Multi-Token Prediction speculative draft heads (`mtp`).

---

## Quantization Ladder

| Filename | Quant Type | Size | Description | Recommended VRAM / RAM |
| :--- | :--- | :--- | :--- | :--- |
| `MiMo-V2.6-Flash-REAP50-MXFP4_MOE.gguf` | **MXFP4_MOE** | 86.06 GiB | Flagship: 1-to-1 native packed MXFP4 experts (32 blk) + BF16 attention/trunk. Exact bit-level fidelity to REAP base. | 96 GiB+ / 1x 128GB Thor or 2x 48GB |
| `MiMo-V2.6-Flash-REAP50-Q2_K.gguf` | **Q2_K** | 61.64 GiB | Optimal Hybrid MoE: sensitive down-projections kept in native MXFP4, gate/up in Q2_K, trunk in Q8_0 (~3.36 BPW). | 64 GiB+ / 3x 24GB GPUs (72GB) or Mac 64-96GB |

### Supporting Towers (Vision, Audio & MTP)

| Filename | Size | Description |
| :--- | :--- | :--- |
| `mmproj-MiMo-V2.6-Flash-REAP50-BF16.gguf` | 2.56 GiB | Multimodal projector (Vision + Audio) in BF16 |
| `mmproj-MiMo-V2.6-Flash-REAP50-Q8_0.gguf` | 1.46 GiB | Multimodal projector (Vision + Audio) quantized to Q8_0 |
| `mtp-MiMo-V2.6-Flash-REAP50-BF16.gguf` | 4.17 GiB | Multi-Token Prediction (MTP) draft head (3 next-n layers) in BF16 |
| `mtp-MiMo-V2.6-Flash-REAP50-Q8_0.gguf` | 2.22 GiB | Multi-Token Prediction (MTP) draft head (3 next-n layers) in Q8_0 |

---

## Key Features

1. **Native MXFP4 MoE Preservation**:
   In the base model, 92.9% of weights are stored as native packed `mxfp4` (32 block size). Our GGUF converter natively repacks these blocks directly into `GGMLQuantizationType.MXFP4`, avoiding costly lossy dequantization cycles while preserving exact native numerical precision.

2. **Multimodal Projectors (`mmproj`)**:
   Xiaomi MiMo-V2.6-Flash incorporates both visual and audio processing towers:
   - Vision encoder (28-layer ViT, 560px patch representation)
   - Audio tokenizer / RVQ speech representations
   Both are packed into standard GGUF multimodal projectors (`mmproj-*-BF16.gguf` and `mmproj-*-Q8_0.gguf`) compatible with `llama.cpp`'s multimodal pipeline.

3. **Multi-Token Prediction (`mtp`)**:
   MiMo-V2.6-Flash includes 3 trained MTP layers for speculative decoding. We ship standalone MTP draft models (`mtp-*-BF16.gguf` and `mtp-*-Q8_0.gguf`) that can be loaded alongside the trunk model with `--draft-model` to accelerate generation.

---

## Running with llama.cpp

### 1. Standard Text Inference (Optimal Hybrid Q2_K)
```bash
./llama-cli \
    -m MiMo-V2.6-Flash-REAP50-Q2_K.gguf \
    -p "You are MiMo, an AI assistant developed by Xiaomi. Explain how MoE expert pruning works:" \
    -n 512 --temp 0.6
```

Or run the flagship bit-for-bit native MXFP4 checkpoint:
```bash
./llama-cli \
    -m MiMo-V2.6-Flash-REAP50-MXFP4_MOE.gguf \
    -p "You are MiMo, an AI assistant developed by Xiaomi. Explain how MoE expert pruning works:" \
    -n 512 --temp 0.6
```

### 2. Speculative Decoding with MTP Draft Head
```bash
./llama-cli \
    -m MiMo-V2.6-Flash-REAP50-Q2_K.gguf \
    --draft-model mtp-MiMo-V2.6-Flash-REAP50-Q8_0.gguf \
    -p "Explain quantum teleportation in detail:" \
    -n 512
```

### 3. Multimodal Inference (Vision & Audio)
```bash
./llama-cli \
    -m MiMo-V2.6-Flash-REAP50-Q2_K.gguf \
    --mmproj mmproj-MiMo-V2.6-Flash-REAP50-Q8_0.gguf \
    --image input.jpg \
    -p "Describe the contents of this image in detail."
```

### 4. OpenAI-Compatible API Server
```bash
./llama-server \
    -m MiMo-V2.6-Flash-REAP50-Q2_K.gguf \
    --mmproj mmproj-MiMo-V2.6-Flash-REAP50-Q8_0.gguf \
    --port 8080 \
    -ngl 99
```

---

## Background & Pruning Method

Pruned using **HOPE** (Higher-Order Pruning of Experts) over a diverse calibration corpus spanning code, math, conversational text, and multimodal reasoning tasks. Rather than relying solely on first-order activation frequencies, HOPE accounts for inter-expert interaction terms:
$$\Delta \mathcal{L} \approx \sum_{i} g_i^T \Delta w_i + \frac{1}{2} \sum_{i,j} \Delta w_i^T H_{ij} \Delta w_j$$
By computing cross-expert Hessian blocks during the calibration pass, 128 experts per layer were optimally selected to minimize perplexity loss under 50% parameter reduction.

---
*Created by [patrickbdevaney](https://huggingface.co/patrickbdevaney).*