README.md

---
pipeline_tag: text-generation
license: other
license_name: other
license_link: https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE
library_name: transformers
base_model: MiniMaxAI/MiniMax-M2.7
tags:
- minimax_m2
- mxfp4_16
- text-generation
- moe
- mixture-of-experts
- custom_code
- 8-bit precision
---

## `mxfp4_16` Quantization of [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7)

**Runtime:** Requires [`tcclaviger/vllm22:latest`](https://hub.docker.com/r/tcclaviger/vllm22) — a **RDNA 4 (gfx12xx)** vLLM image with `mxfp4_16` kernel support. No other vLLM build currently loads these weights.

---

## 1. Introduction

This is an **MXFP4-16** (Mixed-precision 4-bit with 16-element group size) quantized variant of [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7), produced using compressed-tensors with an **IQ4_NL** codebook.

The quantization scheme:
- **Weight bits:** 4-bit per group of 16 elements
- **Codebook:** IQ4_NL (Improved Q4 Normal) — 16 entries, asymmetric, FP4-like scale
- **Target:** All `Linear` layers (MoE experts + FFN + attention projections)
- **Excluded:** Attention `qkv_proj` scales, `block_sparse_moe.gate`, `lm_head`, `embed_tokens`, MTP layers, norms
- **KV cache:** FP8 (e4m3), no dynamic quantization

The result is a model that retains near-BF16 quality while fitting in significantly less VRAM, friendly to high-memory systems (128GB+ unified memory, multi-GPU 4×48 setups, RDNA4/GFX12xx GPUs).

---

## 2. Model Architecture

MiniMax-M2.7 is a **456B-parameter sparse MoE** model with:
- **456B total parameters** (sparse), **~30B activated** per token
- **256 routed experts** per MoE layer, top-8 routing
- **62 transformer layers**
- **3 MTP (Multi-Token Prediction)** layers for speculative decoding
- **200k context window**
- Native tool-calling support

Key architectural details from `config.json`:
- `hidden_size`: 3072, `num_attention_heads`: 48, `num_key_value_heads`: 8, `head_dim`: 128
- `num_local_experts`: 256, `num_experts_per_tok`: 8
- `rope_theta`: 5,000,000, `max_position_embeddings`: 204,800

---

## 3. Quantization Details

### What was quantized

| Layer type | Quantization | Notes |
|---|---|---|
| MoE expert weights (w1/w3/w2) | MXFP4-16, IQ4_NL | Merged `w13_weight_packed` + scales |
| FFN intermediate (gate/up/proj) | MXFP4-16, IQ4_NL | Standard linear layers |
| Attention projections (qkv) | MXFP4-16, IQ4_NL | QKV split handled correctly |

### What was NOT quantized

| Layer | Reason |
|---|---|
| `self_attn.{k,v}_proj` scales | Per-tensor FP16 (no quantization) |
| `block_sparse_moe.gate` | Router — kept BF16 |
| `e_score_correction_bias` | MoE bias — kept BF16 |
| `lm_head` | Output projection — kept BF16 |
| `embed_tokens` | Embedding — kept BF16 |
| MTP layers | Speculative decoding heads — kept BF16 |
| RMSNorm layers | Normalizations — kept BF16 |
| KV cache | FP8 (e4m3), calibrated scales |

### KV Cache

FP8 (e4m3) KV cache is used at runtime (`--kv-cache-dtype fp8_e4m3`). Per-layer scales are calibrated during quantization and stored alongside weights.

---

## 4. Runtime Requirements

### Hardware

- **GPU:** RDNA 4 (gfx12xx) — tested on 4× RX 9700 (RDNA4)
- **Memory:** 128GB+ recommended for long-context workloads
- **OS:** Linux with ROCm support

### Docker Runtime

The **only** validated way to run this model is with the prebuilt RDNA4 vLLM image:

```bash
# Pull the runtime image
docker pull tcclaviger/vllm22:latest

# Run with 8 GPUs
./run-minimax-m2.7-mxfp416.sh <container_name> <port>
```

This image includes:
- Custom Triton attention kernels tuned for RDNA4 (10× faster than ROCm attention at long context)
- Fixed FP8 KV-cache quantization path (2× throughput improvement)
- Tuned GEMM configs for RX 9700
- MXFP4-16 kernels compiled for gfx12xx

---

## 5. Local Deployment

### vLLM (Recommended)

Using the RDNA4 Docker image:

```bash
vllm serve djdeniro/MiniMax-M2.7-MXFP416 \
  --served-model-name minimax-m2.7-mxfp416 \
  --tensor-parallel-size 8 \
  --enable-expert-parallel \
  --disable-cascade-attn \
  --reasoning-parser minimax_m2 \
  --enable-auto-tool-choice \
  --tool-call-parser minimax_m2 \
  --trust-remote-code \
  --gpu-memory-utilization 0.93 \
  --max-model-len 180000 \
  --kv-cache-dtype fp8_e4m3 \
  --attention-backend TRITON_ATTN \
  --override-generation-config '{"max_tokens": 16384}'
```

Or with Docker:

```bash
docker run --name minimax-mxfp416 \
  --rm --tty --ipc=host --shm-size=128g \
  --device /dev/kfd:/dev/kfd \
  --device /dev/dri/renderD128:/dev/dri/renderD128 \
  --device /dev/dri/renderD129:/dev/dri/renderD129 \
  --device /dev/dri/renderD130:/dev/dri/renderD130 \
  --device /dev/dri/renderD132:/dev/dri/renderD132 \
  --device /dev/dri/renderD137:/dev/dri/renderD137 \
  --device /dev/dri/renderD138:/dev/dri/renderD138 \
  --device /dev/dri/renderD139:/dev/dri/renderD139 \
  --device /dev/dri/renderD140:/dev/dri/renderD140 \
  -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
  -e TRUST_REMOTE_CODE=1 \
  -e PYTORCH_TUNABLEOP_ENABLED=1 \
  -e PYTORCH_TUNABLEOP_TUNING=0 \
  -p 8000:8000 \
  tcclaviger/vllm22:latest \
  bash -c "cp /patches/vllm22_minimax_m2.py /app/vllm/vllm/model_executor/models/minimax_m2.py && \
    /app/.venv/bin/pip install -q sentencepiece && \
    exec /app/.venv/bin/vllm serve \
      /app/models/models/vllm/MiniMax-M2.7-MXFP416 \
      --served-model-name minimax-m2.7-mxfp416 \
      --host 0.0.0.0 --port 8000 \
      --trust-remote-code \
      --tensor-parallel-size 8 \
      --disable-cascade-attn \
      --reasoning-parser minimax_m2 \
      --enable-auto-tool-choice --tool-call-parser minimax_m2 \
      --enable-prefix-caching --gpu-memory-utilization 0.93 \
      --max-model-len 180000 --max-num-seqs 48 --max-num-batched-tokens 2048 \
      --kv-cache-dtype fp8_e4m3 \
      --enable-expert-parallel \
      --attention-backend TRITON_ATTN \
      --override-generation-config '{\"max_tokens\": 16384}'"
```

### API Usage (OpenAI-compatible)

```python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY",
)

completion = client.chat.completions.create(
    model="minimax-m2.7-mxfp416",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain what MXFP4 quantization is."}
    ],
    temperature=1.0,
    max_tokens=1024,
)
print(completion.choices[0].message.content)
```

### Tool Calling

MiniMax-M2.7 has native function calling support. Use `reasoning_parser=minimax_m2` and `tool_call_parser=minimax_m2`:

```python
messages = [
    {"role": "user", "content": [
        {"type": "text", "text": "What's the weather in Tokyo?"},
    ]}
]
# The model will generate tool calls with the correct format
```

---

## 6. Chat Template

The model uses a custom Jinja chat template supporting:

- **System messages** with dynamic tool injection
- **Tool calls** in XML format (`<minimax:tool_call>` / `</minimax:tool_call>`)
- **Reasoning content** (`<think>` / `</think>`)
- **Tool responses** with `<response>` XML tags
- **Generation prompts** with thinking prefix

Example with `apply_chat_template`:

```python
from transformers import AutoProcessor, AutoModelForCausalLM

processor = AutoProcessor.from_pretrained(
    "djdeniro/MiniMax-M2.7-MXFP416",
    trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
    "djdeniro/MiniMax-M2.7-MXFP416",
    device_map="auto",
    dtype="auto",
    trust_remote_code=True
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello, how are you?"}
]

inputs = processor.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
output = processor.decode(generated_ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
print(output)
```

---

## 7. Inference Parameters

Recommended defaults:
- `temperature`: 1.0
- `top_p`: 0.95
- `top_k`: 40
- `max_tokens`: 16384 (configurable)

---

## 8. Acknowledgments

- Base model: [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7)
- Quantization inspiration: [tcclaviger/Step-3.7-Flash-240REAP-MXFP416](https://huggingface.co/tcclaviger/Step-3.7-Flash-240REAP-MXFP416)
- Runtime: [tcclaviger/vllm22](https://hub.docker.com/r/tcclaviger/vllm22)

---

## 9. License

This quantized variant inherits the [Apache 2.0 license](https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE) from the base model.