Instructions to use nvidia/Llama-3_3-Nemotron-Super-49B-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use nvidia/Llama-3_3-Nemotron-Super-49B-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="nvidia/Llama-3_3-Nemotron-Super-49B-v1", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("nvidia/Llama-3_3-Nemotron-Super-49B-v1", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use nvidia/Llama-3_3-Nemotron-Super-49B-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "nvidia/Llama-3_3-Nemotron-Super-49B-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Llama-3_3-Nemotron-Super-49B-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1

SGLang

How to use nvidia/Llama-3_3-Nemotron-Super-49B-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "nvidia/Llama-3_3-Nemotron-Super-49B-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Llama-3_3-Nemotron-Super-49B-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "nvidia/Llama-3_3-Nemotron-Super-49B-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "nvidia/Llama-3_3-Nemotron-Super-49B-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use nvidia/Llama-3_3-Nemotron-Super-49B-v1 with Docker Model Runner:
```
docker model run hf.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1
```

Nemotron 253B?

#10

by BoshiAI - opened Apr 4, 2025

Discussion

BoshiAI

Apr 4, 2025

I understand there are supposed to be three models: the 8B, 49B and a 253B model. The other two were released over two weeks ago with the Ultra model listed as 'coming soon'. Is the model still cookin'? Nearly baked? When's it coming out of the oven and sliding it's way onto the bakery track?

I've never been able to run the Llama 405B model, even quantised down all the way I can't get it to run on my 96GB system. I've had to make do with 70B and 123B models, but would love to see how a ~250B model performs. I thought Nemotron was a superb enhancement to base Llama 3, so I'm particulated excited to see what 'Ultra' is like. I've been checking daily in the hopes it might've dropped. Is there an ETA?

itlevy

NVIDIA org Apr 8, 2025

It's here https://huggingface.co/nvidia/Llama-3_1-Nemotron-Ultra-253B-v1, enjoy :)

BoshiAI

Apr 8, 2025

•

edited Apr 8, 2025

Absolutely fantastic! Thank you, I've been waiting with bated breath for this one. :D

My machine (Mac Studio M2 Max 96GB) can support up to about 300B parameters on an IQ2 quant, but there's been nothing interesting over 123B and under under 300B. I know that your enhancements to Llama 3.1 70B had a massive impact on that model. The fact you're enhancing Llama 3.1 400B, whilst shrinking it down to 250B in size so I can run it is quite something.

I like a powerful all-rounder model that is also excellent for role-playing and creative writing, and I know that your enhancements to L3.1 70B were well received by the RP crowd and one of the few models to actually boost overall intelligence over the original to boot.

Just need for folks to GGUF this so I can try it now!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment