Instructions to use nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4")
model = AutoModelForCausalLM.from_pretrained("nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4

SGLang

How to use nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 with Docker Model Runner:
```
docker model run hf.co/nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4
```

Benchmarked on Single GB10 (ASUS GX10)

by andynoodles - opened Aug 12

Discussion

andynoodles

Aug 12

•

edited Aug 12

Summary

Benchmarked speculative decoding variants — baseline vs. DSpark and DFlash, — for nvidia/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-NVFP4 on a GX10 (DGX Spark, GB10, 121GB unified memory), across chat (ShareGPT) and code (HumanEval) workloads. Also checked whether the marlin NVFP4 path actually engages GB10's FP4 tensor cores (spoiler: it doesn't — and can't).

Environment


Hardware	GX10 / DGX Spark (GB10, SM121), 121GB unified LPDDR5x, aarch64
Driver / CUDA	580.142 / 13.0
Server	vLLM `vllm/vllm-openai:v0.27.1`, model-card-recommended DGX Spark settings (`--moe-backend marlin --kv-cache-dtype fp8 --mamba-backend flashinfer --mamba-cache-mode align`)
Constraints	Prefix caching disabled; `--gpu-memory-utilization 0.45` (machine shared with another vLLM server); `--max-model-len 65536`; `--max-num-seqs 64`
Client	`vllm-bench` (Rust rewrite of `vllm bench serve`); `--num-prompts 10` per data point; temp 1.0 / top-p 0.95; 512-token output cap; seed 42
Speculative config	All variants use 3 speculative tokens; draft models are `…-NVFP4-DSpark` (0.8B) / `…-NVFP4-DFlash` (0.7B)

Note: with only 10 prompts, the concurrency = 16 and 32 data points were effectively run at concurrency = 10.

Results — Output tok/s

HumanEval (code)

Variant	c=1	c=2	c=4	c=8	c=16	c=32	Accept rate	Accept length
baseline	77.0	121.7	160.9	198.9	218.7	220.1	—	—
DFlash	100.8	143.1	189.9	239.4	321.1	323.2	52–55%	~2.6
DSpark	126.1	182.3	246.3	317.7	420.0	411.5	75–78%	~3.3

ShareGPT (chat)

Variant	c=1	c=2	c=4	c=8	c=16	c=32	Accept rate	Accept length
baseline	75.3	118.6	156.9	195.4	214.9	216.0	—	—
DFlash	68.2	115.7	151.2	204.4	260.5	261.6	37–38%	~2.1
DSpark	104.7	155.2	200.7	271.6	343.3	334.5	56–61%	~2.8

Single-stream TPOT: baseline 12.8ms → DSpark 7.7ms (HumanEval).

Korgan0

Aug 12

thx. that post helped me to get it running. DSpark fails on vllm 0.26

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment