Instructions to use unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF:UD-Q4_K_XL

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
llama cli -hf unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF:UD-Q4_K_XL

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./llama-cli -hf unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF:UD-Q4_K_XL

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF:UD-Q4_K_XL
# Run inference directly in the terminal:
./build/bin/llama-cli -hf unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF:UD-Q4_K_XL

Use Docker

docker model run hf.co/unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF:UD-Q4_K_XL

LM Studio
Jan

vLLM

How to use unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF:UD-Q4_K_XL

SGLang

How to use unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF with Ollama:
```
ollama run hf.co/unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF:UD-Q4_K_XL
```
Unsloth Desktop
Docker Model Runner
How to use unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF with Docker Model Runner:
```
docker model run hf.co/unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF:UD-Q4_K_XL
```

Lemonade

How to use unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/Llama-3_1-Nemotron-Ultra-253B-v1-GGUF:UD-Q4_K_XL

Run and chat with the model

lemonade run user.Llama-3_1-Nemotron-Ultra-253B-v1-GGUF-UD-Q4_K_XL

List all available models

lemonade list

Atomic Chat

danielhanchen commited on May 26, 2025

Commit

081fa8d

verified ·

1 Parent(s): 6911ec5

Add files using upload-large-folder tool

Browse files

Files changed (20) hide show

.gitattributes +10 -0
IQ4_NL/Llama-3_1-Nemotron-Ultra-253B-v1-IQ4_NL-00001-of-00003.gguf +3 -0
IQ4_NL/Llama-3_1-Nemotron-Ultra-253B-v1-IQ4_NL-00002-of-00003.gguf +3 -0
IQ4_NL/Llama-3_1-Nemotron-Ultra-253B-v1-IQ4_NL-00003-of-00003.gguf +3 -0
IQ4_XS/Llama-3_1-Nemotron-Ultra-253B-v1-IQ4_XS-00001-of-00003.gguf +3 -0
IQ4_XS/Llama-3_1-Nemotron-Ultra-253B-v1-IQ4_XS-00002-of-00003.gguf +3 -0
IQ4_XS/Llama-3_1-Nemotron-Ultra-253B-v1-IQ4_XS-00003-of-00003.gguf +3 -0
Q2_K_L/Llama-3_1-Nemotron-Ultra-253B-v1-Q2_K_L-00001-of-00002.gguf +2 -2
Q2_K_L/Llama-3_1-Nemotron-Ultra-253B-v1-Q2_K_L-00002-of-00002.gguf +2 -2
Q3_K_M/Llama-3_1-Nemotron-Ultra-253B-v1-Q3_K_M-00001-of-00003.gguf +2 -2
Q3_K_M/Llama-3_1-Nemotron-Ultra-253B-v1-Q3_K_M-00002-of-00003.gguf +2 -2
Q3_K_M/Llama-3_1-Nemotron-Ultra-253B-v1-Q3_K_M-00003-of-00003.gguf +2 -2
Q4_1/Llama-3_1-Nemotron-Ultra-253B-v1-Q4_1-00001-of-00004.gguf +3 -0
Q4_1/Llama-3_1-Nemotron-Ultra-253B-v1-Q4_1-00002-of-00004.gguf +3 -0
Q4_1/Llama-3_1-Nemotron-Ultra-253B-v1-Q4_1-00003-of-00004.gguf +3 -0
Q4_1/Llama-3_1-Nemotron-Ultra-253B-v1-Q4_1-00004-of-00004.gguf +3 -0
Q5_K_M/Llama-3_1-Nemotron-Ultra-253B-v1-Q5_K_M-00001-of-00004.gguf +2 -2
Q5_K_M/Llama-3_1-Nemotron-Ultra-253B-v1-Q5_K_M-00002-of-00004.gguf +2 -2
Q5_K_M/Llama-3_1-Nemotron-Ultra-253B-v1-Q5_K_M-00003-of-00004.gguf +2 -2
Q5_K_M/Llama-3_1-Nemotron-Ultra-253B-v1-Q5_K_M-00004-of-00004.gguf +2 -2

@@ -100,3 +100,13 @@ UD-IQ2_M/Llama-3_1-Nemotron-Ultra-253B-v1-UD-IQ2_M-00002-of-00002.gguf filter=lf
 UD-IQ3_XXS/Llama-3_1-Nemotron-Ultra-253B-v1-UD-IQ3_XXS-00002-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
 UD-IQ3_XXS/Llama-3_1-Nemotron-Ultra-253B-v1-UD-IQ3_XXS-00001-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
 UD-IQ2_M/Llama-3_1-Nemotron-Ultra-253B-v1-UD-IQ2_M-00001-of-00002.gguf filter=lfs diff=lfs merge=lfs -text

 UD-IQ3_XXS/Llama-3_1-Nemotron-Ultra-253B-v1-UD-IQ3_XXS-00002-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
 UD-IQ3_XXS/Llama-3_1-Nemotron-Ultra-253B-v1-UD-IQ3_XXS-00001-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
 UD-IQ2_M/Llama-3_1-Nemotron-Ultra-253B-v1-UD-IQ2_M-00001-of-00002.gguf filter=lfs diff=lfs merge=lfs -text
+Q4_1/Llama-3_1-Nemotron-Ultra-253B-v1-Q4_1-00004-of-00004.gguf filter=lfs diff=lfs merge=lfs -text
+IQ4_NL/Llama-3_1-Nemotron-Ultra-253B-v1-IQ4_NL-00003-of-00003.gguf filter=lfs diff=lfs merge=lfs -text
+Q4_1/Llama-3_1-Nemotron-Ultra-253B-v1-Q4_1-00001-of-00004.gguf filter=lfs diff=lfs merge=lfs -text
+IQ4_NL/Llama-3_1-Nemotron-Ultra-253B-v1-IQ4_NL-00001-of-00003.gguf filter=lfs diff=lfs merge=lfs -text
+Q4_1/Llama-3_1-Nemotron-Ultra-253B-v1-Q4_1-00002-of-00004.gguf filter=lfs diff=lfs merge=lfs -text
+Q4_1/Llama-3_1-Nemotron-Ultra-253B-v1-Q4_1-00003-of-00004.gguf filter=lfs diff=lfs merge=lfs -text
+IQ4_NL/Llama-3_1-Nemotron-Ultra-253B-v1-IQ4_NL-00002-of-00003.gguf filter=lfs diff=lfs merge=lfs -text
+IQ4_XS/Llama-3_1-Nemotron-Ultra-253B-v1-IQ4_XS-00003-of-00003.gguf filter=lfs diff=lfs merge=lfs -text
+IQ4_XS/Llama-3_1-Nemotron-Ultra-253B-v1-IQ4_XS-00002-of-00003.gguf filter=lfs diff=lfs merge=lfs -text
+IQ4_XS/Llama-3_1-Nemotron-Ultra-253B-v1-IQ4_XS-00001-of-00003.gguf filter=lfs diff=lfs merge=lfs -text

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:7fd30c58ae47d3df44bb35b68d6796b907e5f3f1127c9a5d0a87ebc5aa78fdab
-size 49911545760

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:df0c2afa573dd653cf8ddedca68a1ea2093317d7f1b119608099f57a2912b608
-size 48429708416

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:54875e63c208a6b383b1652eb4f850205c279b576d4f08ead019cc9743f2e650
-size 49704119648

+version https://git-lfs.github.com/spec/v1
+oid sha256:12615762ad3447710bd047ec205a18808cc17685f4cb3fd7082f9695760a000b
+size 49993820032

+version https://git-lfs.github.com/spec/v1
+oid sha256:b91413c38f33377c937c29bf144de88c23e50c134e95fe3376734d4ab798afe3
+size 49364022592

+version https://git-lfs.github.com/spec/v1
+oid sha256:89cec41cca91df1636afe6aa44bbbf291f701ed043f15be65e46f7d4f10ebb97
+size 43872426400

+version https://git-lfs.github.com/spec/v1
+oid sha256:fe9dba722183246b10ea0790228605a92fefbcc81d8ca392792cb350f77878c5
+size 49865698048

+version https://git-lfs.github.com/spec/v1
+oid sha256:fbe8a8db3d9e2781839f9a8598c33a95f448a943dcc5acca3a027770857e0123
+size 47876944384

+version https://git-lfs.github.com/spec/v1
+oid sha256:e44566be2d15ebeb3b5b999e560891f0fbb0170ffec23c5ecc0a06fae90409d8
+size 37733800288

 version https://git-lfs.github.com/spec/v1
+oid sha256:b440afd572d370f02748964cc7f7655f9b0f1047ea63e755c0039b66d2afc94f
+size 49894148672

 version https://git-lfs.github.com/spec/v1
+oid sha256:e97708ac30bdd7bf8494032fb699f8daebb91a71e9d9650990e10406cfaa82bf
+size 43993443232

 version https://git-lfs.github.com/spec/v1
+oid sha256:63147824cdb70ac125794e7e32c5509a5f96bdc3baca27cb36ab2fa138007951
+size 49788268896

 version https://git-lfs.github.com/spec/v1
+oid sha256:f095a79273ab8e63c76a80319bb930e17ed92c1978324e7faea0f1b86f676c58
+size 48105860256

	@@ -0,0 +1,3 @@

	@@ -0,0 +1,3 @@

	@@ -0,0 +1,3 @@

	@@ -0,0 +1,3 @@

 version https://git-lfs.github.com/spec/v1
+oid sha256:0bba5123b2bc01e796599de57618410f93aac5714e7737ccd7e44454ec03eb32
+size 23990179936

+version https://git-lfs.github.com/spec/v1
+oid sha256:54b1475823b7a255c269edee7f5b83d85091b9845180e4fa07110d0c3b7bb386
+size 49855078880

+version https://git-lfs.github.com/spec/v1
+oid sha256:6c2c001079e237782b3d214b77862ae7b8664a6e88b3fb6d1b2fe497941fd858
+size 47954999776

+version https://git-lfs.github.com/spec/v1
+oid sha256:c318c780bd90a14f3a53b9020c33396306bbd6b4b53ae0ac00f2b07a7e091d37
+size 49797664000

+version https://git-lfs.github.com/spec/v1
+oid sha256:02ff5d020457ed62b945670a2248b421e041e80fa291164d3bfba9a6775a2359
+size 11194404352

 version https://git-lfs.github.com/spec/v1
+oid sha256:881a80592b8732de92b804ac3a020531f8b34795b7be54a37d213d9c3d3a779c
+size 49946074080

 version https://git-lfs.github.com/spec/v1
+oid sha256:108f22124f27f84c0e1b6ae173ff6a4e02439eaa3ae35d5a883f21d335a89da0
+size 49902794048

 version https://git-lfs.github.com/spec/v1
+oid sha256:0ccdf7b06107d413ea95058a05228ff0d95350fa0e5c4e9d55c0f555b2d8ff47
+size 46869188032

 version https://git-lfs.github.com/spec/v1
+oid sha256:d933df5436fe5c58f9a7f388c47a2f8c24a6b32068f8927bb2a859ed69a41de5
+size 31829399552