Instructions to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Use Docker

docker model run hf.co/Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Ollama
How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with Ollama:
```
ollama run hf.co/Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M
```
Unsloth Desktop

How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with Docker Model Runner:
```
docker model run hf.co/Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M
```

Lemonade

How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF / README.md

RhinoWithAcape

Cards: pull references RhinoWithAcape -> Hob-forge org

38cc102 verified about 1 month ago

preview code

Raw

History Blame Contribute Delete

6.2 kB

	---
	license: other
	license_name: nvidia-open-model-agreement
	license_link: https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-agreement/
	base_model: nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16
	tags:
	- nemotron
	- nemotron-3
	- nano
	- omni
	- reasoning
	- mamba2
	- moe
	- hybrid
	- gguf
	- llama.cpp
	- ollama
	- text-only
	- q4_k_m
	- 24gb
	- cpu-offload
	language:
	- en
	pipeline_tag: text-generation
	library_name: gguf
	base_model_relation: quantized
	---
	# Nemotron-3-Nano-Omni-30B-A3B-Reasoning — Text-Only GGUF

	Text-only GGUF conversion of [`nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16`](https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16), released by NVIDIA on 2026-04-28.

	This conversion strips the vision (CRADIO v4-H) and audio (Parakeet) encoders and packages only the language model core for use with `llama.cpp` and Ollama. The text core is a Mamba2-Transformer hybrid Mixture-of-Experts (30B total, 3B active) with reasoning tuning.

	## What this is — and what it isn't

	- ✅ The full text-reasoning capability of Nemotron-3-Nano-Omni, in a single text-only GGUF.
	- ✅ Identical text weights to the omni release (extracted from `language_model.*` tensors, prefix stripped, vision/audio dropped).
	- ❌ Not multimodal. The GGUF cannot accept images, audio, or video. For full omni capability, use NVIDIA's official BF16 / FP8 / NVFP4 weights with a transformers-compatible runtime — multimodal heads in `llama.cpp` would require an upstream PR adding support for CRADIO + Parakeet.
	- ⚠ Distinct from `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16`. That's NVIDIA's text-only sister model. Our weights come from the omni variant's text encoder, which was co-trained with the multimodal heads. Behavior may differ slightly from the standalone text-only release.

	## Architecture

	\| Field \| Value \|
	\|---\|---\|
	\| Architecture \| Mamba2-Transformer hybrid MoE (`NemotronHForCausalLM`) \|
	\| Hybrid pattern \| `MEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEMEME` \|
	\| Parameters \| ~30B total, ~3B active per token \|
	\| Hidden size \| 2688 \|
	\| Layers \| 52 \|
	\| Mamba heads \| 64 \|
	\| Attention heads \| 32 (head_dim 128) \|
	\| Routed experts \| 128 \|
	\| Shared experts \| 1 \|
	\| Top-k routing \| 6 \|
	\| Vocab \| 131,072 \|
	\| Context \| 32K (per `chat_template.jinja`) \|

	## Quants

	This is a 30B-A3B MoE — only 3B params active per token. MoE architectures hold quality well at lower bit widths because routing isolates each token's compute to a small fraction of the model. The Q4_K_M default is the accessible end here; sub-Q4 quants would be wasted disk for quality lost on a model this sparse.

	\| Quant \| Size \| Use case \|
	\|---\|---\|---\|
	\| Q4_K_M \| ~17 GB \| recommended default — accessible end, runs on consumer hardware \|
	\| Q5_K_M \| ~21 GB \| bump quality if you have headroom \|
	\| Q6_K \| ~25 GB \| near-lossless reasoning \|
	\| Q8_0 \| ~32 GB \| reference quality \|
	\| F16 \| ~60 GB \| full precision (uploaded on request — useful for further quantization) \|

	(Sizes approximate — actual sizes confirmed once conversion completes. 30B MoE means total params, not active — disk size scales with total.)

	## Usage — Ollama

	```bash
	# Pull a quant
	huggingface-cli download Hob-forge/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-GGUF \
	Nemotron-3-Nano-Omni-30B-A3B-Reasoning-text-only.Q4_K_M.gguf \
	--local-dir ./nemotron-omni

	# Build local Ollama model
	cd ./nemotron-omni
	cat > Modelfile <<EOF
	FROM ./Nemotron-3-Nano-Omni-30B-A3B-Reasoning-text-only.Q4_K_M.gguf
	PARAMETER num_ctx 32768
	PARAMETER temperature 0.6
	PARAMETER top_p 0.95
	PARAMETER repeat_penalty 1.05
	PARAMETER stop "<\|im_end\|>"
	PARAMETER stop "<\|endoftext\|>"
	EOF
	ollama create nemotron-3-omni-text:Q4_K_M -f Modelfile

	# Use
	ollama run nemotron-3-omni-text:Q4_K_M
	```

	A `Modelfile` is included in this repo.

	### Reasoning toggle

	The chat template supports thinking-mode toggles via tokens in user messages:
	- `/think` — enable thinking
	- `/no_think` — suppress thinking

	You can also pass `"think": false` at the top level of `/api/chat` (NOT inside `options`) to suppress thinking via the API.

	## Usage — llama.cpp

	```bash
	./build/bin/llama-cli \
	-m Nemotron-3-Nano-Omni-30B-A3B-Reasoning-text-only.Q4_K_M.gguf \
	-c 32768 \
	-p "Explain MoE routing in three sentences." \
	-n 256 \
	--temp 0.6
	```

	## License

	Use is governed by the [NVIDIA Open Model Agreement](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-open-model-agreement/). Commercial use is permitted under the agreement's terms. This conversion is a derivative work — same license applies.

	## Conversion details

	- Source: `nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16` downloaded 2026-04-28
	- Tools: `llama.cpp` (commit `1a635cde0`) `convert_hf_to_gguf.py` with the existing `NemotronHForCausalLM` registration (no patches required to the converter)
	- Steps: extracted `language_model.*` tensors → stripped prefix → wrote clean `NemotronHForCausalLM` config → ran converter → quantized

	## Limitations & caveats

	- Text-only. Drop the omni weights if you need vision/audio.
	- Hybrid Mamba2-Attention layers. Some llama.cpp/Ollama features that assume pure-attention models may behave unexpectedly (e.g. context shifting). Standard generation works fine.
	- Brand-new architecture. Released the same day as this conversion. Expect rough edges; please open issues at the discussions tab.
	- Quant quality on hybrid models. Mamba2 layers may be more quant-sensitive than pure attention. If you see degraded reasoning at Q4_K_M, try Q5_K_M or Q6_K.

	## Acknowledgments

	- NVIDIA for the open release of Nemotron-3-Nano-Omni
	- The `llama.cpp` team for `NemotronHForCausalLM` support
	- This conversion produced for the [Zenith swarm](https://huggingface.co/Hob-forge) — autonomous engineering collective project

	## Citation

	If you use this GGUF, please cite NVIDIA's original release:

	```bibtex
	@misc{nvidia2026nemotron3nanoomni,
	title = {Nemotron-3-Nano-Omni-30B-A3B-Reasoning},
	author = {NVIDIA},
	year = {2026},
	month = {April},
	url = {https://huggingface.co/nvidia/Nemotron-3-Nano-Omni-30B-A3B-Reasoning-BF16}
	}
	```