Add SGLang serving instructions

#3
by cm00cm - opened
Files changed (1) hide show
  1. README.md +22 -1
README.md CHANGED
@@ -61,6 +61,7 @@ Our AI models are designed and/or optimized to run on NVIDIA GPU-accelerated sys
61
  ## Software Integration:
62
  **Supported Runtime Engine(s):** <br>
63
  * vLLM <br>
 
64
 
65
  **Supported Hardware Microarchitecture Compatibility:** <br>
66
  * NVIDIA Blackwell <br>
@@ -95,7 +96,7 @@ The model version is NVFP4 1.0 version and is quantized with nvidia-modelopt **v
95
 
96
 
97
  ## Inference:
98
- **Acceleration Engine:** vLLM <br>
99
  **Test Hardware:** NVIDIA Blackwell GB200 <br>
100
 
101
  ## Post Training Quantization
@@ -127,6 +128,26 @@ vllm serve /checkpoint \
127
  --gpu-memory-utilization 0.90
128
  ```
129
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
130
  ## Evaluation
131
  The accuracy benchmark results are presented in the table below:
132
  <table>
 
61
  ## Software Integration:
62
  **Supported Runtime Engine(s):** <br>
63
  * vLLM <br>
64
+ * SGLang <br>
65
 
66
  **Supported Hardware Microarchitecture Compatibility:** <br>
67
  * NVIDIA Blackwell <br>
 
96
 
97
 
98
  ## Inference:
99
+ **Acceleration Engine:** vLLM, SGLang <br>
100
  **Test Hardware:** NVIDIA Blackwell GB200 <br>
101
 
102
  ## Post Training Quantization
 
128
  --gpu-memory-utilization 0.90
129
  ```
130
 
131
+ ### SGLang
132
+
133
+ To serve this checkpoint with [SGLang](https://github.com/sgl-project/sglang) on 4 NVIDIA Blackwell GPUs, run:
134
+
135
+ ```sh
136
+ sglang serve \
137
+ --model-path nvidia/GLM-5.3-Flash-NVFP4 \
138
+ --quantization modelopt_fp4 \
139
+ --tp-size 4 \
140
+ --dsa-prefill-backend trtllm \
141
+ --dsa-decode-backend trtllm \
142
+ --kv-cache-dtype fp8_e4m3 \
143
+ --moe-runner-backend flashinfer_cutlass \
144
+ --reasoning-parser glm45 \
145
+ --tool-call-parser glm47 \
146
+ --mem-fraction-static 0.85 \
147
+ --host 0.0.0.0 \
148
+ --port 30000
149
+ ```
150
+
151
  ## Evaluation
152
  The accuracy benchmark results are presented in the table below:
153
  <table>