Document vLLM 0.25.1 production validation
Browse files
README.md
CHANGED
|
@@ -57,6 +57,7 @@ The production profile uses:
|
|
| 57 |
|
| 58 |
```bash
|
| 59 |
vllm serve WaveCut/gemma-4-26B-A4B-it-heretic-W4A16-G64-RTN-vllm \
|
|
|
|
| 60 |
--served-model-name "Gemma 4 26B Heretic" \
|
| 61 |
--trust-remote-code \
|
| 62 |
--dtype bfloat16 \
|
|
@@ -70,9 +71,18 @@ vllm serve WaveCut/gemma-4-26B-A4B-it-heretic-W4A16-G64-RTN-vllm \
|
|
| 70 |
--tool-call-parser gemma4
|
| 71 |
```
|
| 72 |
|
| 73 |
-
The
|
| 74 |
-
|
| 75 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 76 |
|
| 77 |
## Limitations
|
| 78 |
|
|
|
|
| 57 |
|
| 58 |
```bash
|
| 59 |
vllm serve WaveCut/gemma-4-26B-A4B-it-heretic-W4A16-G64-RTN-vllm \
|
| 60 |
+
--revision 51b3bbe6b47f12a1841e3934ce24e767bb8dec17 \
|
| 61 |
--served-model-name "Gemma 4 26B Heretic" \
|
| 62 |
--trust-remote-code \
|
| 63 |
--dtype bfloat16 \
|
|
|
|
| 71 |
--tool-call-parser gemma4
|
| 72 |
```
|
| 73 |
|
| 74 |
+
The checkpoint was accepted for production on AI Farm with vLLM 0.25.1,
|
| 75 |
+
Torch 2.11.0+cu130, Transformers 5.8.0, and compressed-tensors 0.17.0.
|
| 76 |
+
On an RTX 3090, both vLLM 0.23.0 and 0.25.1 passed 13/13 text and vision case
|
| 77 |
+
groups plus streaming checks. The matrix included exact needle recovery at
|
| 78 |
+
39,186 text tokens and 38,490 multimodal tokens, including concurrency 2.
|
| 79 |
+
Both runtimes also passed 24/24 production-shaped Russian quality probes;
|
| 80 |
+
median probe latency improved from 0.6385 s to 0.6125 s on vLLM 0.25.1.
|
| 81 |
+
|
| 82 |
+
The full environment, timing table, harness, and raw JSONL artifacts are in
|
| 83 |
+
the [AI Farm vLLM report](https://github.com/iamwavecut/aifarm-vllm-openai/blob/af3a2915246a73585c175355af387c23e7360238/results/20260716-vllm-023-vs-0251.md).
|
| 84 |
+
The historical RTX 3090 quantization benchmark and harness remain available
|
| 85 |
+
under `bench/` and `scripts/`.
|
| 86 |
|
| 87 |
## Limitations
|
| 88 |
|