Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,113 @@
+---
+license: mit
+base_model:
+  - XiaomiMiMo/MiMo-V2.6-Flash
+  - patrickbdevaney/MiMo-V2.6-Flash-REAP50
+tags:
+  - gguf
+  - llama.cpp
+  - reap
+  - hope
+  - moe
+  - pruned
+  - multimodal
+  - vision
+  - audio
+  - mtp
+---
+
+# Xiaomi MiMo-V2.6-Flash REAP-50 — GGUF
+
+Official GGUF quantisations of **MiMo-V2.6-Flash-REAP50**, a 50% routed-expert pruned checkpoint of `XiaomiMiMo/MiMo-V2.6-Flash` created with [REAP](https://github.com/CerebrasResearch/reap) and **HOPE** second-order saliency pruning.
+
+* **Base HF Checkpoint**: [patrickbdevaney/MiMo-V2.6-Flash-REAP50](https://huggingface.co/patrickbdevaney/MiMo-V2.6-Flash-REAP50)
+* **Experts Retained**: **128 of 256** routed experts per layer across 47 MoE layers (1 dense layer, 47 MoE layers).
+* **Base Architecture**: Native packed **MXFP4** (`U8`, block size 32) experts with unquantized pure **BF16** attention and embeddings.
+* **Towers Included**: Vision & Audio multimodal projectors (`mmproj`) and Multi-Token Prediction speculative draft heads (`mtp`).
+
+---
+
+## Quantization Ladder
+
+| Filename | Quant Type | Size | Description | Recommended VRAM / RAM |
+| :--- | :--- | :--- | :--- | :--- |
+| `MiMo-V2.6-Flash-REAP50-MXFP4_MOE.gguf` | **MXFP4_MOE** | Pending | Native packed MXFP4 experts (32 blk) + BF16 attention/trunk | 96 GiB+ / 1x 128GB Thor or 2x 48GB |
+| `MiMo-V2.6-Flash-REAP50-Q4_K_M.gguf` | **Q4_K_M** | Pending | 4-bit medium k-quant; optimal quality/speed balance | 64 GiB+ / 2x 3090/4090 or Mac 64GB |
+| `MiMo-V2.6-Flash-REAP50-Q3_K_M.gguf` | **Q3_K_M** | Pending | 3-bit medium k-quant; high compression | 48 GiB+ / 2x 24GB or Mac 48GB |
+| `MiMo-V2.6-Flash-REAP50-IQ4_XS.gguf` | **IQ4_XS** | Pending | 4-bit non-linear importance-quant; small footprint | 56 GiB+ / 2x 32GB or Mac 64GB |
+| `MiMo-V2.6-Flash-REAP50-IQ3_M.gguf` | **IQ3_M** | Pending | 3-bit non-linear importance-quant mixture | 48 GiB+ / 2x 24GB or Mac 48GB |
+
+### Supporting Towers (Vision, Audio & MTP)
+
+| Filename | Size | Description |
+| :--- | :--- | :--- |
+| `mmproj-MiMo-V2.6-Flash-REAP50-BF16.gguf` | 2.56 GiB | Multimodal projector (Vision + Audio) in BF16 |
+| `mmproj-MiMo-V2.6-Flash-REAP50-Q8_0.gguf` | 1.46 GiB | Multimodal projector (Vision + Audio) quantized to Q8_0 |
+| `mtp-MiMo-V2.6-Flash-REAP50-BF16.gguf` | 4.17 GiB | Multi-Token Prediction (MTP) draft head (3 next-n layers) in BF16 |
+| `mtp-MiMo-V2.6-Flash-REAP50-Q8_0.gguf` | 2.22 GiB | Multi-Token Prediction (MTP) draft head (3 next-n layers) in Q8_0 |
+
+---
+
+## Key Features
+
+1. **Native MXFP4 MoE Preservation**:
+   In the base model, 92.9% of weights are stored as native packed `mxfp4` (32 block size). Our GGUF converter natively repacks these blocks directly into `GGMLQuantizationType.MXFP4`, avoiding costly lossy dequantization cycles while preserving exact native numerical precision.
+
+2. **Multimodal Projectors (`mmproj`)**:
+   Xiaomi MiMo-V2.6-Flash incorporates both visual and audio processing towers:
+   - Vision encoder (28-layer ViT, 560px patch representation)
+   - Audio tokenizer / RVQ speech representations
+   Both are packed into standard GGUF multimodal projectors (`mmproj-*-BF16.gguf` and `mmproj-*-Q8_0.gguf`) compatible with `llama.cpp`'s multimodal pipeline.
+
+3. **Multi-Token Prediction (`mtp`)**:
+   MiMo-V2.6-Flash includes 3 trained MTP layers for speculative decoding. We ship standalone MTP draft models (`mtp-*-BF16.gguf` and `mtp-*-Q8_0.gguf`) that can be loaded alongside the trunk model with `--draft-model` to accelerate generation.
+
+---
+
+## Running with llama.cpp
+
+### 1. Standard Text Inference
+```bash
+./llama-cli \
+    -m MiMo-V2.6-Flash-REAP50-Q4_K_M.gguf \
+    -p "You are MiMo, an AI assistant developed by Xiaomi. Explain how MoE expert pruning works:" \
+    -n 512 --temp 0.6
+```
+
+### 2. Speculative Decoding with MTP Draft Head
+```bash
+./llama-cli \
+    -m MiMo-V2.6-Flash-REAP50-Q4_K_M.gguf \
+    --draft-model mtp-MiMo-V2.6-Flash-REAP50-Q8_0.gguf \
+    -p "Explain quantum teleportation in detail:" \
+    -n 512
+```
+
+### 3. Multimodal Inference (Vision & Audio)
+```bash
+./llama-cli \
+    -m MiMo-V2.6-Flash-REAP50-Q4_K_M.gguf \
+    --mmproj mmproj-MiMo-V2.6-Flash-REAP50-Q8_0.gguf \
+    --image input.jpg \
+    -p "Describe the contents of this image in detail."
+```
+
+### 4. OpenAI-Compatible API Server
+```bash
+./llama-server \
+    -m MiMo-V2.6-Flash-REAP50-Q4_K_M.gguf \
+    --mmproj mmproj-MiMo-V2.6-Flash-REAP50-Q8_0.gguf \
+    --port 8080 \
+    -ngl 99
+```
+
+---
+
+## Background & Pruning Method
+
+Pruned using **HOPE** (Higher-Order Pruning of Experts) over a diverse calibration corpus spanning code, math, conversational text, and multimodal reasoning tasks. Rather than relying solely on first-order activation frequencies, HOPE accounts for inter-expert interaction terms:
+$$\Delta \mathcal{L} \approx \sum_{i} g_i^T \Delta w_i + \frac{1}{2} \sum_{i,j} \Delta w_i^T H_{ij} \Delta w_j$$
+By computing cross-expert Hessian blocks during the calibration pass, 128 experts per layer were optimally selected to minimize perplexity loss under 50% parameter reduction.
+
+---
+*Created by [patrickbdevaney](https://huggingface.co/patrickbdevaney).*