Add SGLang serving instructions
Browse filesAdds SGLang as a supported runtime and documents the high-throughput aggregated TP4 serving command from the GLM-5.3-Flash cookbook.
Validation:
- SGLang 0.0.0.dev1+g008403017 on 4x NVIDIA GB300 GPUs
- High-throughput: full GSM8K test split, 1319/1319 examples, 97.19% accuracy, 0 request errors
- Low-latency adaptive EAGLE: full GSM8K test split, 1319/1319 examples, 96.66% accuracy, 0 request errors
This PR intentionally documents the high-throughput command. Addresses discussion #2.
README.md
CHANGED
|
@@ -61,6 +61,7 @@ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated sys
|
|
| 61 |
## Software Integration:
|
| 62 |
**Supported Runtime Engine(s):** <br>
|
| 63 |
* vLLM <br>
|
|
|
|
| 64 |
|
| 65 |
**Supported Hardware Microarchitecture Compatibility:** <br>
|
| 66 |
* NVIDIA Blackwell <br>
|
|
@@ -95,7 +96,7 @@ The model version is NVFP4 1.0 version and is quantized with nvidia-modelopt **v
|
|
| 95 |
|
| 96 |
|
| 97 |
## Inference:
|
| 98 |
-
**Acceleration Engine:** vLLM <br>
|
| 99 |
**Test Hardware:** NVIDIA Blackwell GB200 <br>
|
| 100 |
|
| 101 |
## Post Training Quantization
|
|
@@ -127,6 +128,26 @@ vllm serve /checkpoint \
|
|
| 127 |
--gpu-memory-utilization 0.90
|
| 128 |
```
|
| 129 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 130 |
## Evaluation
|
| 131 |
The accuracy benchmark results are presented in the table below:
|
| 132 |
<table>
|
|
|
|
| 61 |
## Software Integration:
|
| 62 |
**Supported Runtime Engine(s):** <br>
|
| 63 |
* vLLM <br>
|
| 64 |
+
* SGLang <br>
|
| 65 |
|
| 66 |
**Supported Hardware Microarchitecture Compatibility:** <br>
|
| 67 |
* NVIDIA Blackwell <br>
|
|
|
|
| 96 |
|
| 97 |
|
| 98 |
## Inference:
|
| 99 |
+
**Acceleration Engine:** vLLM, SGLang <br>
|
| 100 |
**Test Hardware:** NVIDIA Blackwell GB200 <br>
|
| 101 |
|
| 102 |
## Post Training Quantization
|
|
|
|
| 128 |
--gpu-memory-utilization 0.90
|
| 129 |
```
|
| 130 |
|
| 131 |
+
### SGLang
|
| 132 |
+
|
| 133 |
+
To serve this checkpoint with [SGLang](https://github.com/sgl-project/sglang) on 4 NVIDIA Blackwell GPUs, run:
|
| 134 |
+
|
| 135 |
+
```sh
|
| 136 |
+
sglang serve \
|
| 137 |
+
--model-path nvidia/GLM-5.3-Flash-NVFP4 \
|
| 138 |
+
--quantization modelopt_fp4 \
|
| 139 |
+
--tp-size 4 \
|
| 140 |
+
--dsa-prefill-backend trtllm \
|
| 141 |
+
--dsa-decode-backend trtllm \
|
| 142 |
+
--kv-cache-dtype fp8_e4m3 \
|
| 143 |
+
--moe-runner-backend flashinfer_cutlass \
|
| 144 |
+
--reasoning-parser glm45 \
|
| 145 |
+
--tool-call-parser glm47 \
|
| 146 |
+
--mem-fraction-static 0.85 \
|
| 147 |
+
--host 0.0.0.0 \
|
| 148 |
+
--port 30000
|
| 149 |
+
```
|
| 150 |
+
|
| 151 |
## Evaluation
|
| 152 |
The accuracy benchmark results are presented in the table below:
|
| 153 |
<table>
|