Instructions to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Ollama
How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with Ollama:
```
ollama run hf.co/Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M
```
Unsloth Desktop

How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with Docker Model Runner:
```
docker model run hf.co/Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M
```

Lemonade

How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF / README.md

RhinoWithAcape

Cards: pull references RhinoWithAcape -> Hob-forge org

38cc102 verified about 1 month ago

preview code

Raw

History Blame Contribute Delete

6.2 kB

metadata

license: other
license_name: nvidia-open-model-agreement
license_link: >-
  https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-agreement/
base_model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
tags:
  - nemotron
  - nemotron-3
  - nano
  - omni
  - reasoning
  - mamba2
  - moe
  - hybrid
  - gguf
  - llama.cpp
  - ollama
  - text-only
  - q4_k_m
  - 24gb
  - cpu-offload
language:
  - en
pipeline_tag: text-generation
library_name: gguf
base_model_relation: quantized

Nemotron-3-Nano-Omni-30B-A3B-Reasoning — Text-Only GGUF

Text-only GGUF conversion of nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16, released by NVIDIA on 2026-04-28.

This conversion strips the vision (CRADIO v4-H) and audio (Parakeet) encoders and packages only the language model core for use with llama.cpp and Ollama. The text core is a Mamba2-Transformer hybrid Mixture-of-Experts (30B total, 3B active) with reasoning tuning.

What this is — and what it isn't

✅ The full text-reasoning capability of Nemotron-3-Nano-Omni, in a single text-only GGUF.
✅ Identical text weights to the omni release (extracted from language_model.* tensors, prefix stripped, vision/audio dropped).
❌ Not multimodal. The GGUF cannot accept images, audio, or video. For full omni capability, use NVIDIA's official BF16 / FP8 / NVFP4 weights with a transformers-compatible runtime — multimodal heads in llama.cpp would require an upstream PR adding support for CRADIO + Parakeet.
⚠ Distinct from nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16. That's NVIDIA's text-only sister model. Our weights come from the omni variant's text encoder, which was co-trained with the multimodal heads. Behavior may differ slightly from the standalone text-only release.

Architecture

Field	Value
Architecture	Mamba2-Transformer hybrid MoE (`NemotronHForCausalLM`)
Hybrid pattern	`MEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEME`
Parameters	~30B total, ~3B active per token
Hidden size	2688
Layers	52
Mamba heads	64
Attention heads	32 (head_dim 128)
Routed experts	128
Shared experts	1
Top-k routing	6
Vocab	131,072
Context	32K (per `chat_template.jinja`)

Quants

This is a 30B-A3B MoE — only 3B params active per token. MoE architectures hold quality well at lower bit widths because routing isolates each token's compute to a small fraction of the model. The Q4_K_M default is the accessible end here; sub-Q4 quants would be wasted disk for quality lost on a model this sparse.

Quant	Size	Use case
Q4_K_M	~17 GB	recommended default — accessible end, runs on consumer hardware
Q5_K_M	~21 GB	bump quality if you have headroom
Q6_K	~25 GB	near-lossless reasoning
Q8_0	~32 GB	reference quality
F16	~60 GB	full precision (uploaded on request — useful for further quantization)

(Sizes approximate — actual sizes confirmed once conversion completes. 30B MoE means total params, not active — disk size scales with total.)

Usage — Ollama

# Pull a quant
huggingface-cli download Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF \
    Nemotron-3-Nano-Omni-30B-A3B-Reasoning-text-only.Q4_K_M.gguf \
    --local-dir ./nemotron-omni

# Build local Ollama model
cd ./nemotron-omni
cat > Modelfile <<EOF
FROM ./Nemotron-3-Nano-Omni-30B-A3B-Reasoning-text-only.Q4_K_M.gguf
PARAMETER num_ctx 32768
PARAMETER temperature 0.6
PARAMETER top_p 0.95
PARAMETER repeat_penalty 1.05
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|endoftext|>"
EOF
ollama create nemotron-3-omni-text:Q4_K_M -f Modelfile

# Use
ollama run nemotron-3-omni-text:Q4_K_M

A Modelfile is included in this repo.

Reasoning toggle

The chat template supports thinking-mode toggles via tokens in user messages:

/think — enable thinking
/no_think — suppress thinking

You can also pass "think": false at the top level of /api/chat (NOT inside options) to suppress thinking via the API.

Usage — llama.cpp

./build/bin/llama-cli \
    -m Nemotron-3-Nano-Omni-30B-A3B-Reasoning-text-only.Q4_K_M.gguf \
    -c 32768 \
    -p "Explain MoE routing in three sentences." \
    -n 256 \
    --temp 0.6

License

Use is governed by the NVIDIA Open Model Agreement. Commercial use is permitted under the agreement's terms. This conversion is a derivative work — same license applies.

Conversion details

Source: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16 downloaded 2026-04-28
Tools: llama.cpp (commit 1a635cde0) convert_hf_to_gguf.py with the existing NemotronHForCausalLM registration (no patches required to the converter)
Steps: extracted language_model.* tensors → stripped prefix → wrote clean NemotronHForCausalLM config → ran converter → quantized

Limitations & caveats

Text-only. Drop the omni weights if you need vision/audio.
Hybrid Mamba2-Attention layers. Some llama.cpp/Ollama features that assume pure-attention models may behave unexpectedly (e.g. context shifting). Standard generation works fine.
Brand-new architecture. Released the same day as this conversion. Expect rough edges; please open issues at the discussions tab.
Quant quality on hybrid models. Mamba2 layers may be more quant-sensitive than pure attention. If you see degraded reasoning at Q4_K_M, try Q5_K_M or Q6_K.

Acknowledgments

NVIDIA for the open release of Nemotron-3-Nano-Omni
The llama.cpp team for NemotronHForCausalLM support
This conversion produced for the Zenith swarm — autonomous engineering collective project

Citation

If you use this GGUF, please cite NVIDIA's original release:

@misc{nvidia2026nemotron3nanoomni,
  title  = {Nemotron-3-Nano-Omni-30B-A3B-Reasoning},
  author = {NVIDIA},
  year   = {2026},
  month  = {April},
  url    = {https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16}
}