Instructions to use el4/GRM-3.2-Sky-ONYX-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use el4/GRM-3.2-Sky-ONYX-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf el4/GRM-3.2-Sky-ONYX-GGUF
# Run inference directly in the terminal:
llama cli -hf el4/GRM-3.2-Sky-ONYX-GGUF

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf el4/GRM-3.2-Sky-ONYX-GGUF
# Run inference directly in the terminal:
llama cli -hf el4/GRM-3.2-Sky-ONYX-GGUF

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf el4/GRM-3.2-Sky-ONYX-GGUF
# Run inference directly in the terminal:
./llama-cli -hf el4/GRM-3.2-Sky-ONYX-GGUF

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf el4/GRM-3.2-Sky-ONYX-GGUF
# Run inference directly in the terminal:
./build/bin/llama-cli -hf el4/GRM-3.2-Sky-ONYX-GGUF

Use Docker

docker model run hf.co/el4/GRM-3.2-Sky-ONYX-GGUF

LM Studio
Jan

vLLM

How to use el4/GRM-3.2-Sky-ONYX-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "el4/GRM-3.2-Sky-ONYX-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "el4/GRM-3.2-Sky-ONYX-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/el4/GRM-3.2-Sky-ONYX-GGUF

Ollama
How to use el4/GRM-3.2-Sky-ONYX-GGUF with Ollama:
```
ollama run hf.co/el4/GRM-3.2-Sky-ONYX-GGUF
```
Unsloth Desktop

How to use el4/GRM-3.2-Sky-ONYX-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf el4/GRM-3.2-Sky-ONYX-GGUF

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "el4/GRM-3.2-Sky-ONYX-GGUF"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use el4/GRM-3.2-Sky-ONYX-GGUF with Docker Model Runner:
```
docker model run hf.co/el4/GRM-3.2-Sky-ONYX-GGUF
```

Lemonade

How to use el4/GRM-3.2-Sky-ONYX-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull el4/GRM-3.2-Sky-ONYX-GGUF

Run and chat with the model

lemonade run user.GRM-3.2-Sky-ONYX-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Hermes Agent

How to use el4/GRM-3.2-Sky-ONYX-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf el4/GRM-3.2-Sky-ONYX-GGUF

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default el4/GRM-3.2-Sky-ONYX-GGUF

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use el4/GRM-3.2-Sky-ONYX-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf el4/GRM-3.2-Sky-ONYX-GGUF

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "el4/GRM-3.2-Sky-ONYX-GGUF" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

ollama run hf.co/el4/GRM-3.2-Sky-ONYX-GGUF:GRM-3.2-Sky-ONYX-mini.gguf

To prove this extreme compression didn't lobotomize the model, we subjected the ~11GB ONYX-mini quant to a grueling 20-minute agentic stress test designed to break long-horizon reasoning. We tasked the model with writing a compilable Rust program to calculate quantum harmonic oscillator eigenvalues via nalgebra matrix diagonalization, forcing it to execute the code via bash, read the compiler errors, and autonomously debug its own Unicode and eigenvalue-sorting bugs. After successfully patching the math to achieve <0.05% analytical error, the model seamlessly passed the entire technical explanation through a three-step translation gauntlet into formal Japanese, Spanish, and colloquial Egyptian Arabic. The fact that a heavily crushed 35B MoE running on a consumer laptop GPU can hold the working memory required to write physics code, run cargo build, fix its own mistakes, and perfectly execute multilingual code-switching in just 20 minutes proves that ONYX doesn't just shrink the weights—it preserves the model's agentic soul.

🪨 ONYX

Adaptive Precision Engine

This repo contains ONYX Quants of OrionLLM/GRM-3.2-Sky

⚙️ The ONYX Architecture

🧠 Dynamic Layer Sensitivity

Replaces hardcoded edge boundaries with real activation variance measurements. ONYX autonomously identifies critical layers (like mid-network attention blocks) and protects them dynamically.

🎯 Router-Weighted Imatrix

Captures MoE router probabilities and multiplies them into activation scales. This forces the quantizer to aggressively crush "cold" experts while fiercely protecting "hot" ones within the same tensor block.

🏗️ Architecture-Agnostic

Dynamically reads HuggingFace modules and the generated F16 GGUF to map tensors. No hardcoded regex. Works out-of-the-box on Llama, DeepSeek, and custom hybrid SSM/MoE architectures.

Tier Name	Target Quality	Target Size	Middle Layer Strategy
🪨 `quality`	Q8 Match	~21 GB	IQ4_XS
⚖️ `balanced`	Q6 Match	~24 GB	Q5_K
📦 `compact`	Q4 Match	~16 GB	Q3_K
🚀 `mini`	Q2 Match	~12 GB	IQ2_XXS

📚 Credits & Foundations

👉 APEX Quantization Method
Ettore Di Giacinto & Richard Palethorpe (LocalAI Team). ONYX evolves the layer-wise precision gradients and MoE-aware tensor classification outlined in the APEX technical paper into a fully dynamic, data-driven engine.

👉 Bartowski and Lamim
For the excellent semantic imatrix calibration dataset that powers ONYX's activation scaling.

👉 llama.cpp
Georgi Gerganov and contributors for the foundational inference and quantization engine.

👉 HuggingFace Accelerate
For the init_empty_weights() context manager that makes the 0-RAM "Ghost Model" possible on consumer hardware.

Support the Project

A coffee in Ethereum would be cool! Although I don't drink coffee—I think it tastes like burnt water—but a pink lemonade would be fire! 🔥

0xDEE7fa8C421BD038D32e4441ea1aDe72fE973706

recommended sampling parameters:

--temp 0.85 --top-p 0.95 --top-k 40 --min-p 0.05 --repeat-penalty 1.1 --presence-penalty 0.0

Downloads last month: 19,804

GGUF

Model size

35B params

Architecture

qwen35moe

Hardware compatibility

We're not able to determine the quantization variants.

View all variants

Model tree for el4/GRM-3.2-Sky-ONYX-GGUF

Base model

ornith-ai/Ornith-1.0-35B

Finetuned

OrionLLM/GRM-3.2-Sky

Quantized

(16)

this model

Collection including el4/GRM-3.2-Sky-ONYX-GGUF

ONYX

Collection

6 items • Updated 1 day ago • 3