Commit ·
070513c
1
Parent(s): d9eb85d
docs: update SGLang quickstart (#1)
Browse files- docs: update SGLang quickstart (fec9b26a370afbc43d5a68d00fdf4adee1d6022d)
README.md
CHANGED
|
@@ -338,7 +338,45 @@ print(message.content)
|
|
| 338 |
|
| 339 |
</details>
|
| 340 |
|
| 341 |
-
### Option B -
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 342 |
|
| 343 |
The checkpoint can also be loaded directly from the Hugging Face repository.
|
| 344 |
Its `configuration_k2_horizon.py` and `modeling_k2_horizon.py` files are loaded
|
|
|
|
| 338 |
|
| 339 |
</details>
|
| 340 |
|
| 341 |
+
### Option B - SGLang (native architecture support)
|
| 342 |
+
|
| 343 |
+
Native K2 Horizon support is provided by
|
| 344 |
+
[sgl-project/sglang#37654](https://github.com/sgl-project/sglang/pull/37654).
|
| 345 |
+
Use a `lmsysorg/sglang:dev` image built after that PR is merged. Once support is
|
| 346 |
+
included in a tagged SGLang release, use the corresponding versioned image.
|
| 347 |
+
|
| 348 |
+
<details>
|
| 349 |
+
<summary>Show the SGLang serving command</summary>
|
| 350 |
+
|
| 351 |
+
```bash
|
| 352 |
+
docker run --gpus all \
|
| 353 |
+
--shm-size 32g \
|
| 354 |
+
-p 30000:30000 \
|
| 355 |
+
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
| 356 |
+
--ipc=host \
|
| 357 |
+
lmsysorg/sglang:dev \
|
| 358 |
+
python3 -m sglang.launch_server \
|
| 359 |
+
--model-path "IFM/K2-Horizon-0.9B" \
|
| 360 |
+
--revision main \
|
| 361 |
+
--tp 1 \
|
| 362 |
+
--dtype bfloat16 \
|
| 363 |
+
--context-length 8192 \
|
| 364 |
+
--attention-backend fa3 \
|
| 365 |
+
--reasoning-parser k2_horizon \
|
| 366 |
+
--tool-call-parser k2_horizon \
|
| 367 |
+
--mem-fraction-static 0.85 \
|
| 368 |
+
--host 0.0.0.0 \
|
| 369 |
+
--port 30000
|
| 370 |
+
```
|
| 371 |
+
|
| 372 |
+
</details>
|
| 373 |
+
|
| 374 |
+
This uses SGLang's native `K2HorizonForCausalLM` implementation; no
|
| 375 |
+
`--trust-remote-code`, source patch, or external parser plugin is required. The
|
| 376 |
+
8,192-token limit is a conservative starting point; increase it only after
|
| 377 |
+
measuring KV-cache capacity.
|
| 378 |
+
|
| 379 |
+
### Option C - plain Transformers (no vLLM, no container)
|
| 380 |
|
| 381 |
The checkpoint can also be loaded directly from the Hugging Face repository.
|
| 382 |
Its `configuration_k2_horizon.py` and `modeling_k2_horizon.py` files are loaded
|