Add production vllm serve example (TP4, tool calling, MTP spec decode)
Browse files
README.md
CHANGED
|
@@ -28,10 +28,35 @@ using **AutoRound** via [llm-compressor](https://github.com/vllm-project/llm-com
|
|
| 28 |
|
| 29 |
## Usage (vLLM ≥ 0.26)
|
| 30 |
|
|
|
|
|
|
|
|
|
|
| 31 |
```bash
|
| 32 |
-
|
| 33 |
-
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
```
|
| 36 |
|
| 37 |
- Vision inputs work normally (image/video); the vision tower runs in BF16.
|
|
|
|
| 28 |
|
| 29 |
## Usage (vLLM ≥ 0.26)
|
| 30 |
|
| 31 |
+
Production example (4× RTX 3090, tool calling + reasoning parser + MTP speculative decoding).
|
| 32 |
+
Replace the local path with the repo id `DoktorMincs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-W4A16-AutoRound` to load directly from the Hub:
|
| 33 |
+
|
| 34 |
```bash
|
| 35 |
+
export CUDA_DEVICE_ORDER=PCI_BUS_ID
|
| 36 |
+
export OMP_NUM_THREADS=4
|
| 37 |
+
export CUDA_VISIBLE_DEVICES=0,1,2,3
|
| 38 |
+
export PYTORCH_ALLOC_CONF=expandable_segments:True
|
| 39 |
+
export NCCL_P2P_DISABLE=1
|
| 40 |
+
export FLASHINFER_DISABLE_VERSION_CHECK=1
|
| 41 |
+
|
| 42 |
+
exec vllm serve \
|
| 43 |
+
/root/models/DoktorMincs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-W4A16-AutoRound \
|
| 44 |
+
--served-model-name "qwen3.6-27b" \
|
| 45 |
+
--tensor-parallel-size 4 \
|
| 46 |
+
--max-model-len 163840 \
|
| 47 |
+
--override-generation-config '{"temperature": 0.6, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0, "repetition_penalty": 1.0}' \
|
| 48 |
+
--tool-call-parser qwen3_coder \
|
| 49 |
+
--reasoning-parser qwen3 \
|
| 50 |
+
--enable-auto-tool-choice \
|
| 51 |
+
--trust-remote-code \
|
| 52 |
+
--enable-prefix-caching \
|
| 53 |
+
--max-num-batched-tokens 8192 \
|
| 54 |
+
--gpu-memory-utilization 0.88 \
|
| 55 |
+
--disable-custom-all-reduce \
|
| 56 |
+
--max-num-seqs 16 \
|
| 57 |
+
--host 0.0.0.0 \
|
| 58 |
+
--port 3434 \
|
| 59 |
+
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'
|
| 60 |
```
|
| 61 |
|
| 62 |
- Vision inputs work normally (image/video); the vision tower runs in BF16.
|