Does vllm/vllm-openai:nightly work for you?

Yep, only nightly is working right now.

I've asked about official image because in playbook for dgx to run nvfp4 models nvidia suggests to use their image and I was confusing to get an error

解决了吗
我也遇到同样的问题了

with nighty version , how much token/s ? i try a lot model, none of them satified me, i use Qwen3.6-27b-fp8 now. poor speed for generation.

For generating 90-180t/s on practice. Depends on kv cache.

Right now it's very good model using vllm v0.23.0

With preserve_thinking it's lower and I don't suggest to use preserve_thinking for this model.

can you share the config for .sh ,i try a lot , can't reach even 20 token/s

can you share the config for .sh ,i try a lot , can't reach even 20 token/s

export CUTE_DSL_ARCH=sm_121a
export FLASHINFER_DISABLE_VERSION_CHECK=1
export VLLM_USE_FLASHINFER_MOE_FP4=0
export VLLM_FP8_MOE_BACKEND=flashinfer_cutlass

docker run -it --gpus all -p 8000:8000 \
  -e HF_TOKEN="<your-token>" \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --ipc=host \
  vllm/vllm-openai:v0.23.0 \
  --model nvidia/Qwen3.6-35B-A3B-NVFP4 \
  --port 8000 \
  --reasoning-parser qwen3 \
  --dtype auto \
  --kv-cache-dtype fp8 \
  --tensor-parallel-size 1 \
  --max-num-seqs 4 \
  --max-model-len 262000 \
  --gpu-memory-utilization 0.6 \
  --trust-remote-code \
  --enable-prefix-caching \
  --enable-prompt-tokens-details \
  --max-num-batched-tokens 32768 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --quantization modelopt \
  --attention-backend flashinfer \
  --moe-backend marlin \
  --enable-chunked-prefill \
  --async-scheduling \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' \
  --default-chat-template-kwargs '{"preserve_thinking":false}'

anyway, thx a lot, i'l test it, but , i am sure that it gonna be a lot of problem like sm121a issue for my dgx-spark

But I'm also using this model on dgx spark gb10 and it's working very well. I'm using config above to run it

ok ,i use the script you provide, it works, i found that i use the wrong version of vllm. i pull image with vllm/vllm-openai:v0.23.0-cu129. better than qwen3.6-27b. anyway, thanks a lot.

Pleased to report I'm getting up to 80 tps on a Gigabyte AI Top Atom 128GB (DGX Spark) (0:
Here's my sh config

docker run --runtime nvidia --gpus all \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    -p 8000:8000 \
    --ipc=host \
    -e HF_TOKEN=$HF_TOKEN \
    -e VLLM_NVFP4_GEMM_BACKEND=flashinfer-cutlass \
    vllm/vllm-openai:latest \
    --model nvidia/Qwen3.6-35B-A3B-NVFP4 \
    --quantization modelopt \
    --kv-cache-dtype fp8_e4m3 \
    --attention-backend flashinfer \
    --moe-backend marlin \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.85 \
    --max-model-len 131072 \
    --trust-remote-code \
    --reasoning-parser qwen3 \
    --enable-auto-tool-choice \
    --tool-call-parser qwen3_xml

thx a lot, i will try it to see it is ok for my machine. last configuration already can run, but i feel not as others' speed. thx again.

This is final advice from claude opus 5:

You're done chasing — that's roughly the hardware's ceiling.
Where you landed: ~20–30 tok/s steady state, spec decode accepting 71–90% (up from 38% earlier), env vars all confirmed applied.
For reference, a careful benchmark of the FP8 version of this same model on a DGX Spark measured 28–30 tok/s single-stream. You're at parity. The 97 and 174 tok/s figures I cited earlier are from less rigorous sources and I'd treat them as decode-only numbers that exclude real request overhead.
GB10's memory bandwidth is ~273 GB/s. Decode is bandwidth-bound. That's the wall.
What's actually left
The MoE kernel tuning (steps 5–9 from before) is worth ~10%. Real but modest — 25 → ~28 tok/s.
Concurrency is where the headroom is. Single-stream is fixed, but aggregate scales well: that same benchmark got 70 tok/s at 4 concurrent, 156 at 32. Your --max-num-seqs 8 supports this already — if your workload can batch, you get 3–5x more total throughput for free.
Leave num_speculative_tokens at 3. Your per-position rates (0.931, 0.792) show the drafter earning its keep now.
One thing worth testing if you want more
The GB10 optimization notes claim quantizing lm_head to 4-bit is worth +47% on bandwidth-bound hardware — it's read every single decode step. That's model surgery, not a flag, and it means re-quantizing the checkpoint. Biggest remaining lever by far, but a project rather than a restart.
Sources:
Qwen3.6-35B-A3B-FP8 benchmark on DGX Spark GB10
gb10-dgx-spark-llm-optimizations

The following here is the configuration i have optimized(start_service.sh):

#!/usr/bin/env bash

Qwen3.6-35B-A3B-NVFP4 on DGX Spark (GB10 / sm_121a) via vLLM

set -euo pipefail

MODEL_DIR=/home/my-userid/llm_models
MODEL=/llm_models/Qwen3.6-35B-A3B-NVFP4

NVFP4 on sm_121 needs v0.24.0+. Pin it -- never use :latest for a working deployment.

IMAGE=vllm/vllm-openai:v0.24.0-cu129

docker rm -f vllm-qwen36 2>/dev/null || true

docker run -itd
--name vllm-qwen36
--init
--gpus all
--ipc=host
-p 8000:8000
-v "${MODEL_DIR}":/llm_models:ro
# --- GB10 / sm_121 specific ---
-e CUTE_DSL_ARCH=sm_121a
-e FLASHINFER_DISABLE_VERSION_CHECK=1
-e PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True
-e VLLM_MARLIN_USE_ATOMIC_ADD=1
-e VLLM_NVFP4_GEMM_BACKEND=marlin
-e HF_HUB_OFFLINE=1
"${IMAGE}"
--model "${MODEL}"
--served-model-name qwen3.6-35b qwen3.6-27b
--port 8000
# --- quantization / memory ---
--quantization modelopt
--kv-cache-dtype fp8
--dtype auto
--safetensors-load-strategy=prefetch
--tensor-parallel-size 1
# 0.85 OOMs: vLLM reports "Estimated CUDA graph memory: -20 GiB" on GB10,
# which inflates the KV budget by ~20 GiB. 0.6 absorbs the error.
--gpu-memory-utilization 0.6
--max-model-len 262144
--trust-remote-code
# --- kernels ---
--attention-backend flashinfer
# marlin is the only working NVFP4 path on sm_121 (no native FP4 tensor cores)
--moe-backend marlin
# --- throughput ---
--enable-prefix-caching
--enable-chunked-prefill
--async-scheduling
--max-num-seqs 8
--max-num-batched-tokens 16384
--speculative-config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}'
# --- API behaviour ---
--reasoning-parser qwen3
--enable-auto-tool-choice
--tool-call-parser qwen3_coder
--enable-prompt-tokens-details
--default-chat-template-kwargs '{"preserve_thinking":false}'

echo "Waiting for server..."
until curl -sf http://localhost:8000/v1/models >/dev/null; do sleep 5; done
echo "Ready: http://localhost:8000/v1"

The total memory using around 84.6G out of 122G