Great daily driver on 2x5060Ti 16GB
I've been using this for coding with opencode pretty much daily. This is my vllm config
services:
vllm:
image: docker.io/vllm/vllm-openai:qwen38
ports:
- "8080:8000"
volumes:
- ./models:/models:ro
- ./cache:/root/.cache
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ['0','1']
capabilities: [gpu]
shm_size: '45gb'
ulimits:
memlock:
soft: -1
hard: -1
command: >
/models/QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4
--served-model-name qwen38
--api-key "bigsecret"
--tensor-parallel-size 2
--max-model-len 191608
--max-num-seqs 2
--max_num_batched_tokens 3072
--enable-chunked-prefill
--gpu-memory-utilization 0.954
--kv-cache-dtype fp8
--enable-prefix-caching
--trust-remote-code
--enable-auto-tool-choice
--tool-call-parser qwen3_xml
--reasoning-parser qwen3
--language-model-only
--default-chat-template-kwargs '{"enable_thinking": true,"preserve_thinking": true,"reasoning_effort":"medium"}'
--override-generation-config '{"temperature":1.0,"top_p":0.95,"top_k":20,"min_p":0.0,"repetition_penalty":1.0}'
--enable-prompt-tokens-details
--enable-per-request-metrics
--kv_offloading_backend native
--kv_offloading_size 36
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3}'
--mamba-cache-dtype bfloat16
I get between 35 and 60t/s when generating and the vllm caching keeps it humming along. So thanks and well done.
Agreed! I've been using it for a couple of weeks now as an endpoint for my Openclaw and Hermes. My goal was to stretch out the max length on dual 5060ti's (16gb each) with VLLM, while fulfilling the requirements of MTP + NVFP4, vision model left out.
export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
export VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS="fp4_gemm"
Activate vLLM virtual environment
source /home/user/venvs/vllm/bin/activate
exec vllm serve QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4
--tensor-parallel-size 2
--kv-cache-dtype fp8
--gpu-memory-utilization 0.90
--max-model-len 200000
--max-num-seqs 1
--max-num-batched-tokens 2048
--enable-chunked-prefill
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--enable-prefix-caching
--language-model-only
--speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'
I also range mid 60~t/s, dropping down to low 30's at max context of my limit, to which I consider as high efficiency.
$ python test-ai-curl.py
=======================================================
MTP Speculative Decoding Performance (k = 2)
Total Speculative Steps: 52,248
Total Tokens Drafted: 104,494
Total Tokens Accepted: 80,222
Overall Acceptance Rate: 76.77%
Effective Tokens / Step: 2.54
Position | Accepted | Absolute Hit Rate | Retention
Pos 0 | 43168 | 82.62% | 100.0%
Pos 1 | 37054 | 70.92% | 85.8%
This is great to hear, thanks a lot for sharing the config and numbers! 35-60 tok/s on 2x 5060 Ti 16GB is really nice especially with that context length.
Also really appreciate you posting the full vLLM setup here