# vLLM serving (opt-in)

`serve_vllm.py` serves blink-4b text-only through `/v1/systemone`, `/v1/models` and `/healthz`. `serve.py` stays the default; it can take screenshots when enabled.

From the model folder at revision `v1.4`:

Tested versions:

```sh
python -m pip install "torch==2.13.0" "transformers==5.17.0" \
  "vllm==0.30.0" "compressed-tensors==0.17.0" \
  "accelerate>=1.1.0" safetensors huggingface_hub
python -m pip install "flash-linear-attention==0.5.2"
```

```sh
python serve_vllm.py --model . --port 8000 \
  --quantization auto --max-concurrency 32 --max-num-seqs 32 \
  --max-num-batched-tokens 8192 --gpu-memory-utilization 0.85
```

Tested: vLLM 0.30.0; Transformers 5.17.0; Torch 2.13.0; compressed-tensors 0.17.0; flash-linear-attention 0.5.2. BF16 weights, FP32 offered-label head; chunked prefill on, prefix caching off.

Tested vLLM context: `--max-model-len 32768`; on this server, a question over 32,768 tokens returns `422`. Default `serve.py` allows 131,072 tokens per question; another vLLM context size needs a new quality check.

## Quality

The tables are from the earlier E3 study; capacity was not retimed and the other quality sets and c16 were not rerun. A release build re-passed the 231-item JevBench c1 quality check (80/111 hard, 199/231 total, hard ECE 0.069). Later changes touched only request checks, error handling and startup cleanup, not scoring.

| Task measure | vLLM result | Acceptance |
| --- | ---: | ---: |
| JevBench public hard correct | 80/111 | at least 78/111 |
| JevBench public total correct | 199/231 | at least 196/231 |
| JevBench public hard top-label ECE (rounded) | c1 0.069; c16 first 0.069 / repeat 0.074 | at most ~0.077 on each |
| Web actions, 5 options (c1): agreement with the same model's FP32 answer | 499/500 | at least 489/500 |
| Web actions, 9 options (c1): agreement with the same model's FP32 answer | 494/500 | at least 487/500 |
| Decision Index 0.1 latency-tail sample (160 requests, 1,851 questions): agreement with the same model's FP32 answer | c1 1846/1851; c16 first 1847/1851 / repeat 1849/1851 | at least 1836/1851 on each |

Both web-action sets have 500 text-only questions each from public Multimodal-Mind2Web; ECE is rounded here, but checked unrounded. The Decision Index 0.1 latency-tail sample is 160 length-selected requests (1,851 questions) from reference-covered u1000, excluding source mismatches; it is not full u1000, DI-S or 0.2.

## Capacity

| Measure | `serve.py` control | `serve_vllm.py` |
| --- | ---: | ---: |
| JevBench c1 p50 / p95 | 64 / 175 ms | 52 / 127 ms |
| Longest TypeSafe documents (16 requests), c1 p95 | 22.2 s | 16.4 s |
| JevBench c16 completed q/s | 12.2 | 34.7 |
| Decision Index 0.1 latency-tail sample (160 requests, 1,851 questions), c16 completed q/s | 21.0 | 46.5 |
| Peak device memory while scoring | 21.4 GiB | 68.0 GiB |
| Startup through health and warm score | first not captured; later control under 19 s | first cold start 180 s |

One self-hosted replica, loopback HTTP. c1 uses one request at a time; c16 uses 16 concurrent requests and reports completed questions/s. Peak memory includes reserved cache. Quality was checked on these samples, not on every Decision Index item. Agreement with the same model's FP32 answer is not gold accuracy or bit-for-bit parity. For more traffic, use separate warm replicas; recheck quality after changing flags or runtime. No official JevBench score for blink has been published.

Any request with a top-level `images` field (even `[]`) or inline image data returns `422` (`"this model reads text only"`); the server reports `accepts_images: false`. `--quantization auto` keeps these BF16 weights. The INT8 builds that were tried did not pass the quality checks.

Code: Apache-2.0. Weights: non-commercial research and evaluation only; see `LICENSE.md`.