Instructions to use nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16", trust_remote_code=True, device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16

SGLang

How to use nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16 with Docker Model Runner:
```
docker model run hf.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16
```

doesn't do kv caching on transformers

#14

by adaface-neurips - opened Dec 18, 2025

Discussion

adaface-neurips

Dec 18, 2025

If use_cache=True, I got a warning:
NemotronH requires an initialized NemotronHHybridDynamicCache to return a cache. None was provided, so no cache will be returned.
The speed is around 1-2 tokens/s on H200. So it seems kv cache is not enabled due to this warning.
I tried to monkey patch the code (e.g. using past_key_values as cache_params), but the code always runs into various errors, so I gave up.
Any suggestions? Can I only do fast inference using vllm?
Thanks.

suhara

NVIDIA org Dec 18, 2025

Hi @adaface-neurips

Thank you for the message. We're aware that the current HF implementation has an issue with KV/MambaCache and are trying to address it.

Please note that HF is primarily for prototyping and other inference engines (vLLM, TRT-LLM, SGLang, Llama.cpp) optimized for Nemotron 3 Nano are available and support KV Cache. Please check Quick Start and try other inference engines meanwhile.

adaface-neurips

Dec 19, 2025

Thank you @suhara for your reply!
I understand that other inference engines are recommended. If I wish to finetune nemotron (e.g. with RL on math datasets), what framework would you recommend to use for training? Thank you very much again.

dmax123

NVIDIA org May 5

Hi @adaface-neurips
NemoRL supports fine-tuning nemotron-3-nano. Please see doc here
Also, the nemotron modeling code has been upstreamed to transformers lib. Please upgrade transformers to v5.3.0+. You don't need to add trust_remote_code=True when loading the model

taylorseries

May 25

Hi @adaface-neurips
NemoRL supports fine-tuning nemotron-3-nano. Please see doc here
Also, the nemotron modeling code has been upstreamed to transformers lib. Please upgrade transformers to v5.3.0+. You don't need to add trust_remote_code=True when loading the model

I'm using Transformer 5.4.0, but the following issue still persists. Could you please let me know what the reason might be?
WARNING:transformers_modules._1.modeling_nemotron_h:NemotronH requires an initialized `NemotronHHybridDynamicCache` to return a cache. None was provided, so no cache will be returned.

Alexxuxu

Jun 24

Hi @adaface-neurips
NemoRL supports fine-tuning nemotron-3-nano. Please see doc here
Also, the nemotron modeling code has been upstreamed to transformers lib. Please upgrade transformers to v5.3.0+. You don't need to add trust_remote_code=True when loading the model

I'm using Transformer 5.4.0, but the following issue still persists. Could you please let me know what the reason might be?
WARNING:transformers_modules._1.modeling_nemotron_h:NemotronH requires an initialized `NemotronHHybridDynamicCache` to return a cache. None was provided, so no cache will be returned.

fix?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment