DoktorMincs commited on
Commit
e311f07
·
verified ·
1 Parent(s): 9eb8150

Add production vllm serve example (TP4, tool calling, MTP spec decode)

Browse files
Files changed (1) hide show
  1. README.md +28 -3
README.md CHANGED
@@ -28,10 +28,35 @@ using **AutoRound** via [llm-compressor](https://github.com/vllm-project/llm-com
28
 
29
  ## Usage (vLLM ≥ 0.26)
30
 
 
 
 
31
  ```bash
32
- vllm serve DoktorMincs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-W4A16-AutoRound \
33
- --tensor-parallel-size 2 --max-model-len 65536 \
34
- --speculative-config '{"method":"mtp","num_speculative_tokens":1}'
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
35
  ```
36
 
37
  - Vision inputs work normally (image/video); the vision tower runs in BF16.
 
28
 
29
  ## Usage (vLLM ≥ 0.26)
30
 
31
+ Production example (4× RTX 3090, tool calling + reasoning parser + MTP speculative decoding).
32
+ Replace the local path with the repo id `DoktorMincs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-W4A16-AutoRound` to load directly from the Hub:
33
+
34
  ```bash
35
+ export CUDA_DEVICE_ORDER=PCI_BUS_ID
36
+ export OMP_NUM_THREADS=4
37
+ export CUDA_VISIBLE_DEVICES=0,1,2,3
38
+ export PYTORCH_ALLOC_CONF=expandable_segments:True
39
+ export NCCL_P2P_DISABLE=1
40
+ export FLASHINFER_DISABLE_VERSION_CHECK=1
41
+
42
+ exec vllm serve \
43
+ /root/models/DoktorMincs/Qwen3.6-27B-Fable-Fusion-711-Uncensored-Heretic-NM-DAU-MTP-W4A16-AutoRound \
44
+ --served-model-name "qwen3.6-27b" \
45
+ --tensor-parallel-size 4 \
46
+ --max-model-len 163840 \
47
+ --override-generation-config '{"temperature": 0.6, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0, "repetition_penalty": 1.0}' \
48
+ --tool-call-parser qwen3_coder \
49
+ --reasoning-parser qwen3 \
50
+ --enable-auto-tool-choice \
51
+ --trust-remote-code \
52
+ --enable-prefix-caching \
53
+ --max-num-batched-tokens 8192 \
54
+ --gpu-memory-utilization 0.88 \
55
+ --disable-custom-all-reduce \
56
+ --max-num-seqs 16 \
57
+ --host 0.0.0.0 \
58
+ --port 3434 \
59
+ --speculative-config '{"method": "mtp", "num_speculative_tokens": 1}'
60
  ```
61
 
62
  - Vision inputs work normally (image/video); the vision tower runs in BF16.