Default max-num-seqs=16, fp8-KV 128K serve example, measured concurrency (~10 full-128K sessions)

README.md

@@ -86,13 +86,19 @@ answer in `content`. Omit it if you want the raw text (think tags included).
 CUDA_VISIBLE_DEVICES=0 vllm serve sakamakismile/LFM2.5-8B-A1B-NVFP4 \
     --served-model-name lfm25-8b-a1b \
     --quantization modelopt \
-    --max-model-len 32768 \
-    --max-num-seqs 8 \
+    --kv-cache-dtype fp8 \
+    --max-model-len 128000 \
+    --max-num-seqs 16 \
     --gpu-memory-utilization 0.90 \
     --reasoning-parser deepseek_r1 \
     --port 8000
 ```
 
+**Concurrency on one 16 GB card (measured):** with `--kv-cache-dtype fp8` the KV pool is
+**1.38 M tokens** → **~10 sessions each at a full 128 K context** (bf16 KV → ~5). With
+paged KV and shorter prompts you can serve far more — `--max-num-seqs 16` is a good
+default; pin to `10` to guarantee every slot at full 128 K.
+
 | flag | what it does for *this* model |
 |---|---|
 | `--quantization modelopt` | **required** — reads `hf_quant_config.json` (NVFP4). Omit it and weights load as raw uint8 → garbage. |

USAGE.md

@@ -11,8 +11,9 @@ Tested: vLLM 0.21, CUDA 12.8/13.x, SM120. ~117 tok/s single, ~326 tok/s @4 concu
 CUDA_VISIBLE_DEVICES=0 vllm serve /path/to/LFM2.5-8B-A1B-NVFP4 \
     --served-model-name lfm25-8b-a1b \
     --quantization modelopt \
-    --max-model-len 32768 \
-    --max-num-seqs 8 \
+    --kv-cache-dtype fp8 \
+    --max-model-len 128000 \
+    --max-num-seqs 16 \
     --gpu-memory-utilization 0.90 \
     --reasoning-parser deepseek_r1 \
     --port 8000
@@ -34,7 +35,7 @@ curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -
 |---|---|---|
 | `--quantization modelopt` | **required** | tells vLLM the checkpoint is NVFP4 (`hf_quant_config.json`). Without it vLLM treats weights as raw uint8 → garbage. |
 | `--max-model-len` | `32768` (up to `128000`) | context window. KV is cheap here (only 6 of 24 layers carry KV), so you can go long or run many sessions. |
-| `--max-num-seqs` | `8` (try `16`+) | concurrent sequences. Small weights (6.9 GB) leave ~7.5 GB for KV ≈ **658 K tokens**, so this card holds *a lot* of parallel sessions — raise it for throughput. |
+| `--max-num-seqs` | `16` | concurrent sequences. Measured on a 16 GB card: with `--kv-cache-dtype fp8` the KV pool is **1.38 M tokens** → **~10 sessions each at a FULL 128 K context** (bf16 KV → ~5). 16 leaves headroom for shorter sessions via paged KV; pin to `10` if you want every slot guaranteed at full 128 K. |
 | `--gpu-memory-utilization` | `0.90` | fraction of VRAM vLLM may use. 0.90 is safe on a clean 16 GB card; lower if the card is shared. |
 | `--reasoning-parser deepseek_r1` | recommended | LFM2.5 is a **reasoning model** emitting `<think>…</think>` then the answer. This parser puts the CoT in `reasoning_content` and the answer in `content`. Omit it to get the raw text (think tags included). |
 | `--kv-cache-dtype fp8` | optional | ~doubles KV capacity (→ ~1.3 M tokens) for even more concurrency; tiny quality cost. Default (omit) keeps bf16 KV. |

compose.yaml

@@ -13,7 +13,7 @@ services:
       - MODEL_DIR=/model
       - PORT=8000
       - MAX_MODEL_LEN=${MAX_MODEL_LEN:-32768}
-      - MAX_NUM_SEQS=${MAX_NUM_SEQS:-8}
+      - MAX_NUM_SEQS=${MAX_NUM_SEQS:-16}
       - GPU_MEM_UTIL=${GPU_MEM_UTIL:-0.90}
       - KV_CACHE_DTYPE=${KV_CACHE_DTYPE:-auto}
       - TP_SIZE=${TP_SIZE:-1}

entrypoint.sh

@@ -8,7 +8,7 @@ HOST="${HOST:-0.0.0.0}"
 
 # --- tunables (all env-overridable) ---
 MAX_MODEL_LEN="${MAX_MODEL_LEN:-32768}"     # up to 128000
-MAX_NUM_SEQS="${MAX_NUM_SEQS:-8}"           # raise for more concurrency (KV is cheap here)
+MAX_NUM_SEQS="${MAX_NUM_SEQS:-16}"          # ~10 full-128K足軽 + headroom for shorter ones (fp8 KV)
 GPU_MEM_UTIL="${GPU_MEM_UTIL:-0.90}"
 KV_CACHE_DTYPE="${KV_CACHE_DTYPE:-auto}"    # set "fp8" to ~double KV capacity
 TP_SIZE="${TP_SIZE:-1}"                     # this model fits 1 GPU — keep 1