Add RDNA4/RX9700 vLLM deployment guide

docs/vllm_deploy_guide.md

@@ -0,0 +1,193 @@
+# MiniMax-M2.7-MXFP416 vLLM Deployment Guide (RDNA 4 / RX 9700)
+
+## Hardware Requirements
+
+- **GPU:** AMD RX 9700 (RDNA 4 / gfx12xx) — minimum 4x recommended
+- **Memory:** 128GB+ system RAM for 8-GPU setup
+- **OS:** Linux with ROCm 6.x
+- **Docker:** RDNA4-compatible vLLM image
+
+## Docker Image
+
+The only validated runtime for this model is [`tcclaviger/vllm22:latest`](https://hub.docker.com/r/tcclaviger/vllm22):
+
+```bash
+docker pull tcclaviger/vllm22:latest
+```
+
+This image includes:
+- Custom Triton attention kernels tuned for RDNA4 (significantly faster than ROCm attention at long context)
+- Fixed FP8 KV-cache quantization path (~2× throughput improvement)
+- Pre-tuned GEMM configs for RX 9700
+- MXFP4-16 kernels compiled for gfx12xx
+
+## System Setup
+
+### GPU Devices
+
+Make sure all GPUs are visible:
+
+```bash
+rocm-smi --showid
+# Should show: 0, 1, 2, 3, 4, 5, 6, 7
+```
+
+### Power Limit (Recommended)
+
+RDNA4 performs best with tuned power limits. Default is ~300W but 210W provides better sustained throughput on multi-GPU setups:
+
+```bash
+# Set per-GPU power limit
+for i in 0 1 2 3 4 5 6 7; do
+    rocm-smi --setpowerlimit $i 210
+done
+```
+
+> **Note:** At full power (300W) sustained speeds are lower due to thermal throttling. At 210W, sustained generation throughput is consistently higher under multi-user workloads.
+
+## Launching the Server
+
+### Single Container (8 GPUs)
+
+```bash
+docker run --name minimax-mxfp416 \
+  --rm --tty --ipc=host --shm-size=128g \
+  --device /dev/kfd:/dev/kfd \
+  --device /dev/dri/renderD128:/dev/dri/renderD128 \
+  --device /dev/dri/renderD129:/dev/dri/renderD129 \
+  --device /dev/dri/renderD130:/dev/dri/renderD130 \
+  --device /dev/dri/renderD132:/dev/dri/renderD132 \
+  --device /dev/dri/renderD137:/dev/dri/renderD137 \
+  --device /dev/dri/renderD138:/dev/dri/renderD138 \
+  --device /dev/dri/renderD139:/dev/dri/renderD139 \
+  --device /dev/dri/renderD140:/dev/dri/renderD140 \
+  -e HIP_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
+  -e ROCR_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 \
+  -e TRUST_REMOTE_CODE=1 \
+  -e PYTORCH_TUNABLEOP_ENABLED=1 \
+  -e PYTORCH_TUNABLEOP_TUNING=0 \
+  -e PYTORCH_TUNABLEOP_RECORD_UNTUNED=0 \
+  -e PYTORCH_ALLOC_CONF=expandable_segments:True \
+  -e PYTORCH_HIP_ALLOC_CONF=expandable_segments:True \
+  -e GPU_MAX_HW_QUEUES=1 \
+  -v /path/to/models:/app/models:ro \
+  -v /path/to/patches:/patches:ro \
+  -p 8000:8000 \
+  tcclaviger/vllm22:latest \
+  bash -c "cp /patches/vllm22_minimax_m2.py /app/vllm/vllm/model_executor/models/minimax_m2.py && \
+    pip install -q sentencepiece && \
+    exec vllm serve \
+      /app/models/MiniMax-M2.7-MXFP416 \
+      --served-model-name minimax-m2.7-mxfp416 \
+      --host 0.0.0.0 --port 8000 \
+      --trust-remote-code \
+      --tensor-parallel-size 8 \
+      --enable-expert-parallel \
+      --disable-cascade-attn \
+      --reasoning-parser minimax_m2 \
+      --enable-auto-tool-choice \
+      --tool-call-parser minimax_m2 \
+      --enable-prefix-caching \
+      --gpu-memory-utilization 0.93 \
+      --max-model-len 180000 \
+      --max-num-seqs 48 \
+      --max-num-batched-tokens 2048 \
+      --kv-cache-dtype fp8_e4m3 \
+      --attention-backend TRITON_ATTN \
+      --override-generation-config '{\"max_tokens\": 16384}'"
+```
+
+### Key vLLM Flags Explained
+
+| Flag | Value | Purpose |
+|------|-------|---------|
+| `--tensor-parallel-size` | 8 | Split model across 8 GPUs |
+| `--enable-expert-parallel` | | Enable expert-parallel distribution |
+| `--disable-cascade-attn` | | Disable cascade attention for MoE layers |
+| `--attention-backend` | TRITON_ATTN | Use Triton kernels (10× faster than ROCm on RDNA4) |
+| `--kv-cache-dtype` | fp8_e4m3 | FP8 KV cache (~50% memory savings) |
+| `--enable-prefix-caching` | | Cache common prefixes (93%+ hit rate observed) |
+| `--max-model-len` | 180000 | 180k context |
+| `--max-num-seqs` | 48 | Max concurrent sequences |
+| `--max-num-batched-tokens` | 2048 | Max tokens per batch |
+| `--gpu-memory-utilization` | 0.93 | Use 93% of GPU memory |
+
+> **Important:** The `--disable-cascade-attn` flag is required for MoE models. Without it, the model will produce incorrect outputs.
+
+### Running with Patches
+
+If you have custom model patches:
+
+```bash
+-v /path/to/patches:/patches:ro \
+```
+
+The Docker entry point copies `vllm22_minimax_m2.py` to the vLLM model directory before launching. This adds MXFP4-16 support for MiniMax-M2.7.
+
+## Performance Notes
+
+### Observed Performance (4× RX 9700, 210W power limit)
+
+- **Generation throughput:** 50–80 tokens/s
+- **Prefill throughput:** 2000+ tokens/s (with prefix caching)
+- **Prefix cache hit rate:** ~93%
+- **KV cache usage:** 25–33% typical at 180k context
+- **Max concurrent users:** 4–5 at full 180k context
+
+### KV Cache Capacity
+
+With 8× RX 9700 and FP8 KV cache:
+- **KV cache memory:** 11.35 GiB
+- **KV cache tokens:** ~768K tokens
+- **Max context per request:** 180,000 tokens
+- **Max concurrent at 180k:** ~4 requests
+
+### Model Loading
+
+- **Weight loading time:** ~42 seconds
+- **Memory per GPU (TP8):** ~17.5 GiB
+- **Torch compile warmup:** ~37 seconds
+
+## Testing the Deployment
+
+```python
+from openai import OpenAI
+
+client = OpenAI(
+    base_url="http://localhost:8000/v1",
+    api_key="EMPTY",
+)
+
+completion = client.chat.completions.create(
+    model="minimax-m2.7-mxfp416",
+    messages=[
+        {"role": "system", "content": "You are a helpful assistant."},
+        {"role": "user", "content": "Explain what MXFP4 quantization is in one sentence."}
+    ],
+    temperature=1.0,
+    max_tokens=256,
+)
+
+print(completion.choices[0].message.content)
+```
+
+## Troubleshooting
+
+### "expandable_segments not supported"
+
+This warning is benign on ROCm. The model runs correctly despite the warning.
+
+### Low throughput at long context
+
+Ensure `TRITON_ATTN` backend is active. Default ROCm attention is 10× slower on RDNA4 at long context.
+
+### Thermal throttling
+
+If sustained throughput degrades over time, reduce power limit to 210W per GPU:
+```bash
+rocm-smi --setpowerlimit 0 210
+```
+
+### Model fails to load
+
+Ensure `--trust-remote-code` is set and the model path is correct. The custom model file (`vllm22_minimax_m2.py`) must be copied before vLLM loads the model.