Update README with RDNA4 deployment guide and performance stats

README.md

@@ -25,115 +25,45 @@ tags:
 
 This is an **MXFP4-16** (Mixed-precision 4-bit with 16-element group size) quantized variant of [MiniMaxAI/MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7), produced using compressed-tensors with an **IQ4_NL** codebook.
 
-The quantization scheme:
-- **Weight bits:** 4-bit per group of 16 elements
-- **Codebook:** IQ4_NL (Improved Q4 Normal) — 16 entries, asymmetric, FP4-like scale
-- **Target:** All `Linear` layers (MoE experts + FFN + attention projections)
-- **Excluded:** Attention `qkv_proj` scales, `block_sparse_moe.gate`, `lm_head`, `embed_tokens`, MTP layers, norms
-- **KV cache:** FP8 (e4m3), no dynamic quantization
+The quantization:
+- **4-bit** weights with 16-element group size, IQ4_NL codebook
+- All `Linear` layers quantized (MoE experts, FFN, attention projections)
+- Attention `k/v_proj` scales, router gate, MTP layers, norms, embeddings kept BF16
+- KV cache: FP8 (e4m3), calibrated scales baked into checkpoint
 
-The result is a model that retains near-BF16 quality while fitting in significantly less VRAM, friendly to high-memory systems (128GB+ unified memory, multi-GPU 4×48 setups, RDNA4/GFX12xx GPUs).
+The result fits in ~17.5 GiB per GPU (TP8) while retaining near-BF16 quality.
 
 ---
 
 ## 2. Model Architecture
 
-MiniMax-M2.7 is a **456B-parameter sparse MoE** model with:
-- **456B total parameters** (sparse), **~30B activated** per token
-- **256 routed experts** per MoE layer, top-8 routing
-- **62 transformer layers**
-- **3 MTP (Multi-Token Prediction)** layers for speculative decoding
+- **456B total params** (sparse), **~30B activated** per token
+- **256 experts** per MoE layer, top-8 routing, 62 transformer layers
+- **3 MTP layers** for speculative decoding
 - **200k context window**
 - Native tool-calling support
 
-Key architectural details from `config.json`:
-- `hidden_size`: 3072, `num_attention_heads`: 48, `num_key_value_heads`: 8, `head_dim`: 128
-- `num_local_experts`: 256, `num_experts_per_tok`: 8
-- `rope_theta`: 5,000,000, `max_position_embeddings`: 204,800
-
----
-
-## 3. Quantization Details
-
-### What was quantized
-
-| Layer type | Quantization | Notes |
-|---|---|---|
-| MoE expert weights (w1/w3/w2) | MXFP4-16, IQ4_NL | Merged `w13_weight_packed` + scales |
-| FFN intermediate (gate/up/proj) | MXFP4-16, IQ4_NL | Standard linear layers |
-| Attention projections (qkv) | MXFP4-16, IQ4_NL | QKV split handled correctly |
-
-### What was NOT quantized
-
-| Layer | Reason |
-|---|---|
-| `self_attn.{k,v}_proj` scales | Per-tensor FP16 (no quantization) |
-| `block_sparse_moe.gate` | Router — kept BF16 |
-| `e_score_correction_bias` | MoE bias — kept BF16 |
-| `lm_head` | Output projection — kept BF16 |
-| `embed_tokens` | Embedding — kept BF16 |
-| MTP layers | Speculative decoding heads — kept BF16 |
-| RMSNorm layers | Normalizations — kept BF16 |
-| KV cache | FP8 (e4m3), calibrated scales |
-
-### KV Cache
-
-FP8 (e4m3) KV cache is used at runtime (`--kv-cache-dtype fp8_e4m3`). Per-layer scales are calibrated during quantization and stored alongside weights.
-
 ---
 
-## 4. Runtime Requirements
+## 3. Runtime Requirements
 
-### Hardware
+- **GPU:** RDNA 4 (gfx12xx) — 4× or 8× RX 9700 recommended
+- **Memory:** 128GB+ system RAM
+- **Docker:** `tcclaviger/vllm22:latest` — only validated runtime
 
-- **GPU:** RDNA 4 (gfx12xx) — tested on 4× RX 9700 (RDNA4)
-- **Memory:** 128GB+ recommended for long-context workloads
-- **OS:** Linux with ROCm support
-
-### Docker Runtime
-
-The **only** validated way to run this model is with the prebuilt RDNA4 vLLM image:
-
-```bash
-# Pull the runtime image
-docker pull tcclaviger/vllm22:latest
-
-# Run with 8 GPUs
-./run-minimax-m2.7-mxfp416.sh <container_name> <port>
-```
-
-This image includes:
-- Custom Triton attention kernels tuned for RDNA4 (10× faster than ROCm attention at long context)
-- Fixed FP8 KV-cache quantization path (2× throughput improvement)
-- Tuned GEMM configs for RX 9700
-- MXFP4-16 kernels compiled for gfx12xx
+The Docker image includes:
+- Custom Triton attention kernels tuned for RDNA4
+- Fixed FP8 KV-cache quantization path
+- Pre-tuned GEMM configs for RX 9700
+- MXFP4-16 kernels for gfx12xx
 
 ---
 
-## 5. Local Deployment
+## 4. Deployment
 
-### vLLM (Recommended)
+**Full deployment guide (RDNA4 / RX 9700):** [docs/vllm_deploy_guide.md](./docs/vllm_deploy_guide.md)
 
-Using the RDNA4 Docker image:
-
-```bash
-vllm serve djdeniro/MiniMax-M2.7-MXFP416 \
-  --served-model-name minimax-m2.7-mxfp416 \
-  --tensor-parallel-size 8 \
-  --enable-expert-parallel \
-  --disable-cascade-attn \
-  --reasoning-parser minimax_m2 \
-  --enable-auto-tool-choice \
-  --tool-call-parser minimax_m2 \
-  --trust-remote-code \
-  --gpu-memory-utilization 0.93 \
-  --max-model-len 180000 \
-  --kv-cache-dtype fp8_e4m3 \
-  --attention-backend TRITON_ATTN \
-  --override-generation-config '{"max_tokens": 16384}'
-```
-
-Or with Docker:
+Quick-start:
 
 ```bash
 docker run --name minimax-mxfp416 \
@@ -150,44 +80,54 @@ docker run --name minimax-mxfp416 \
   -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
   -e ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
   -e TRUST_REMOTE_CODE=1 \
-  -e PYTORCH_TUNABLEOP_ENABLED=1 \
-  -e PYTORCH_TUNABLEOP_TUNING=0 \
+  -v /path/to/models:/app/models:ro \
+  -v /path/to/patches:/patches:ro \
   -p 8000:8000 \
   tcclaviger/vllm22:latest \
   bash -c "cp /patches/vllm22_minimax_m2.py /app/vllm/vllm/model_executor/models/minimax_m2.py && \
-    /app/.venv/bin/pip install -q sentencepiece && \
-    exec /app/.venv/bin/vllm serve \
-      /app/models/models/vllm/MiniMax-M2.7-MXFP416 \
+    pip install -q sentencepiece && \
+    exec vllm serve /app/models/MiniMax-M2.7-MXFP416 \
       --served-model-name minimax-m2.7-mxfp416 \
-      --host 0.0.0.0 --port 8000 \
-      --trust-remote-code \
-      --tensor-parallel-size 8 \
+      --host 0.0.0.0 --port 8000 --trust-remote-code \
+      --tensor-parallel-size 8 --enable-expert-parallel \
       --disable-cascade-attn \
       --reasoning-parser minimax_m2 \
       --enable-auto-tool-choice --tool-call-parser minimax_m2 \
       --enable-prefix-caching --gpu-memory-utilization 0.93 \
       --max-model-len 180000 --max-num-seqs 48 --max-num-batched-tokens 2048 \
-      --kv-cache-dtype fp8_e4m3 \
-      --enable-expert-parallel \
-      --attention-backend TRITON_ATTN \
+      --kv-cache-dtype fp8_e4m3 --attention-backend TRITON_ATTN \
       --override-generation-config '{\"max_tokens\": 16384}'"
 ```
 
-### API Usage (OpenAI-compatible)
+### Performance (8× RX 9700, 210W power limit)
+
+| Metric | Value |
+|--------|-------|
+| Generation throughput | ~50–80 tokens/s |
+| Prefill throughput | up to 2,190 tokens/s (w/ prefix cache) |
+| Prefix cache hit rate | ~93% |
+| KV cache memory | 11.35 GiB |
+| KV cache capacity | 767,856 tokens |
+| Max context per request | 180,000 tokens |
+| Max concurrent (180k) | 4 requests |
+| Model weight memory (TP8) | ~17.5 GiB/GPU |
+
+> **Power tip:** Set `rocm-smi --setpowerlimit <i> 210` per GPU. At 210W sustained throughput is higher than at full 300W due to reduced thermal throttling.
+
+---
+
+## 5. API Usage
 
 ```python
 from openai import OpenAI
 
-client = OpenAI(
-    base_url="http://localhost:8000/v1",
-    api_key="EMPTY",
-)
+client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
 
 completion = client.chat.completions.create(
     model="minimax-m2.7-mxfp416",
     messages=[
         {"role": "system", "content": "You are a helpful assistant."},
-        {"role": "user", "content": "Explain what MXFP4 quantization is."}
+        {"role": "user", "content": "Hello!"}
     ],
     temperature=1.0,
     max_tokens=1024,
@@ -195,74 +135,43 @@ completion = client.chat.completions.create(
 print(completion.choices[0].message.content)
 ```
 
-### Tool Calling
-
-MiniMax-M2.7 has native function calling support. Use `reasoning_parser=minimax_m2` and `tool_call_parser=minimax_m2`:
-
-```python
-messages = [
-    {"role": "user", "content": [
-        {"type": "text", "text": "What's the weather in Tokyo?"},
-    ]}
-]
-# The model will generate tool calls with the correct format
-```
-
 ---
 
 ## 6. Chat Template
 
-The model uses a custom Jinja chat template supporting:
-
-- **System messages** with dynamic tool injection
-- **Tool calls** in XML format (`<minimax:tool_call>` / `</minimax:tool_call>`)
-- **Reasoning content** (`<think>` / `</think>`)
-- **Tool responses** with `<response>` XML tags
-- **Generation prompts** with thinking prefix
-
-Example with `apply_chat_template`:
+The model uses a Jinja chat template supporting system messages, tool calls (`<minimax:tool_call>`/`</minimax:tool_call>`), reasoning content (`<think>`/`</think>`), and tool responses (`<response>`).
 
 ```python
 from transformers import AutoProcessor, AutoModelForCausalLM
 
 processor = AutoProcessor.from_pretrained(
-    "djdeniro/MiniMax-M2.7-MXFP416",
-    trust_remote_code=True
+    "djdeniro/MiniMax-M2.7-MXFP416", trust_remote_code=True
 )
 model = AutoModelForCausalLM.from_pretrained(
     "djdeniro/MiniMax-M2.7-MXFP416",
-    device_map="auto",
-    dtype="auto",
-    trust_remote_code=True
+    device_map="auto", dtype="auto", trust_remote_code=True
 )
 
 messages = [
     {"role": "system", "content": "You are a helpful assistant."},
     {"role": "user", "content": "Hello, how are you?"}
 ]
-
 inputs = processor.apply_chat_template(
-    messages,
-    tokenize=True,
-    add_generation_prompt=True,
-    return_dict=True,
-    return_tensors="pt",
+    messages, tokenize=True, add_generation_prompt=True,
+    return_dict=True, return_tensors="pt"
 ).to(model.device)
-
-generated_ids = model.generate(**inputs, max_new_tokens=128, do_sample=False)
-output = processor.decode(generated_ids[0][inputs.input_ids.shape[1]:], skip_special_tokens=True)
-print(output)
+out = model.generate(**inputs, max_new_tokens=128, do_sample=False)
+print(processor.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
 ```
 
 ---
 
 ## 7. Inference Parameters
 
-Recommended defaults:
 - `temperature`: 1.0
 - `top_p`: 0.95
 - `top_k`: 40
-- `max_tokens`: 16384 (configurable)
+- `max_tokens`: 16384 (default)
 
 ---
 
@@ -276,4 +185,4 @@ Recommended defaults:
 
 ## 9. License
 
-This quantized variant inherits the [Apache 2.0 license](https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE) from the base model.
+Apache 2.0 — inherits from [base model](https://github.com/MiniMax-AI/MiniMax-M2.7/blob/main/LICENSE).