Can't run on dgx spark with vllm
I was trying to run this model on my dgx spark using suggested vllm command from the model card for NVFP4.
And it's overloading memory with 240k context and the system is stuck! When it's loading it's using 74.8GiB so why does it using the whole memory even if I've set gpu-memory-utilization to 0.9?
I've tried nightly and 0.22.0 versions.
My command:
docker run --rm -it --gpus all \
-e VLLM_NVFP4_GEMM_BACKEND=marlin \
-e VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
-e VLLM_FLASHINFER_ALLREDUCE_BACKEND=trtllm \
-e VLLM_USE_FLASHINFER_MOE_FP4=0 \
-e HF_TOKEN="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" \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-v $(pwd)/super_v3_reasoning_parser.py:/app/super_v3_reasoning_parser.py \
-p 8000:8000 \
vllm/vllm-openai:latest \
--model nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 \
--served-model-name nvidia/nemotron-3-super \
--host 0.0.0.0 \
--port 8000 \
--async-scheduling \
--dtype auto \
--kv-cache-dtype fp8 \
--tensor-parallel-size 1 \
--pipeline-parallel-size 1 \
--data-parallel-size 1 \
--trust-remote-code \
--gpu-memory-utilization 0.90 \
--enable-chunked-prefill \
--max-num-seqs 4 \
--max-model-len 240000 \
--max-num-batched-tokens 16384 \
--moe-backend marlin \
--mamba_ssm_cache_dtype float16 \
--quantization fp4 \
--speculative_config '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}' \
--reasoning-parser-plugin /app/super_v3_reasoning_parser.py \
--reasoning-parser super_v3 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
Event if I'm setting gpu-utilization to 0.7 - it's overloading with 200k max_model_len.
No, I can't run it with provided configs and lower settings on dgx spark. It's overloading to 130Gb VRAM and dropped. So it's not usable on a single dgx spark.
It's running fine on my DGX Spark with VLLM in docker.
But it just can't stop thinking. It's crazy.
Here is the compose file.
services:
vllm-nemotron-super:
image: vllm/vllm-openai:v0.27.1
container_name: vllm-nemotron-super
restart: "no" # switch to unless-stopped once you've confirmed it's stable
ports:
- "8002:8000"
volumes:
- hf-cache:/root/.cache/huggingface
- ./super_v3_reasoning_parser.py:/app/super_v3_reasoning_parser.py:ro
environment:
HF_TOKEN: "${HF_TOKEN}" # pull from your SOPS-decrypted .env, don't hardcode
VLLM_NVFP4_GEMM_BACKEND: marlin
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
VLLM_FLASHINFER_ALLREDUCE_BACKEND: trtllm
VLLM_USE_FLASHINFER_MOE_FP4: "0"
command:
- --model=nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4
- --served-model-name=nvidia/nemotron-3-super
- --host=0.0.0.0
- --port=8000
- --async-scheduling
- --dtype=auto
- --kv-cache-dtype=fp8
- --tensor-parallel-size=1
- --pipeline-parallel-size=1
- --data-parallel-size=1
- --trust-remote-code
- --gpu-memory-utilization=0.90
- --enable-chunked-prefill
- --max-num-seqs=4
- --max-model-len=1000000
- --moe-backend=marlin
- --mamba_ssm_cache_dtype=float16
- --quantization=fp4
# --- add back once base config is confirmed stable ---
# - --speculative_config={"method":"mtp","num_speculative_tokens":3,"model":"nvidia/Nemotron-3-Super-120B-A12B-BF16-MTPv2","moe_backend":"triton"}
- --reasoning-parser-plugin=/app/super_v3_reasoning_parser.py
- --reasoning-parser=super_v3
- --enable-auto-tool-choice
- --tool-call-parser=qwen3_coder
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
shm_size: "16gb"
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/v1/models"]
interval: 30s
timeout: 10s
retries: 20
start_period: 900s # 120B on GB10 will take longer to load than Lightning did
volumes: