Instructions to use ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M

Use Docker

docker model run hf.co/ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M

LM Studio
Jan

vLLM

How to use ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M

Ollama
How to use ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor with Ollama:
```
ollama run hf.co/ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M
```
Unsloth Desktop

How to use ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor with Docker Model Runner:
```
docker model run hf.co/ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M
```

Lemonade

How to use ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M

Run and chat with the model

lemonade run user.gemma-4-E2B-it-AB-GGUF-Editor-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

`gemma-4-E2B-it-AB` — Precision-Optimized GGUF Quantization

Engineered exclusively with GGUF Editor D++
Custom RMSE/MAD optimization | Per-tensor quantization mapping | Maximum quality/size ratio

This repository contains a scientifically optimized quantization of Google's Gemma 4 E4B-it. Unlike standard auto-quantizers, every tensor was individually analyzed and mapped to a specific quantization type to preserve critical reasoning pathways while minimizing memory footprint. The entire pipeline runs on GGUF Editor D++ with custom presets pushed to their theoretical limits.

⚙️ Quantization Pipeline & Custom Parameters

The weights were generated through a multi-stage optimization process within GGUF Editor D++:

Algorithm Pushed to Limits: Utilized the internal Impl mode with highly customized RMSE search parameters (Rmin, Rdelta, Nstep tuned for dense grid coverage across all sensitivity tiers).
Error Smoothing: UseMAD (Mean Absolute Deviation) was enabled to flatten error distribution, preventing outlier-induced degradation in attention heads and MLP gates.
Per-Tensor Mapping: Instead of uniform bit-depths, each tensor was assigned the optimal quantization type (Q4_S, Q4_M, Q5_S, Q5_M, etc.) based on its statistical sensitivity and contribution to model coherence.
Real-Time Validation: All conversions were verified using the built-in Tensor Visualizer to ensure distribution stability and detect anomalies before export.

📊 Comparative Analysis & Error Metrics

One of the primary goals of this release was to demonstrate the superiority of the GGUF Editor D++ (GED++) quantization engine compared to standard implementations (Google's default and Unsloth).

📉 Quantization Fidelity (RMSE/MAE Analysis)

Our testing shows that the "Impl" mode in GGUF Editor D++ provides a significantly cleaner quantization than the standard llama.cpp approach used by other providers.

Implementation	MAE (Mean Absolute Error)	RMS (Root Mean Square)	Accuracy Note
Google (Standard Q4_0)	`0.000774`	`0.001906`	High error/loss
Unsloth (RMSE Optimized)	`0.000088`	`0.000147`	Good precision
GGUF Editor D++ (Our Build)	`0.000056`	`0.000108`	Superior Fidelity

Visual comparison of error distribution and weight stability.

📈 Quality vs. Size Hierarchy

Internal benchmarking confirms a clear quality gradient across quantization tiers. The hierarchy follows the established rule: Q6 > Q5 > Q4 in terms of perplexity stability, logical coherence, and instruction-following accuracy.

Variant	Quantization	Quality Retention	File Size	Best Use Case
`gemma-4-E2B-it-AB-Q6_K_M`	`Q6_K_M`	⭐⭐⭐⭐⭐ (95-100%)	~4.13 GB	Research, maximum reasoning fidelity
`gemma-4-E2B-it-AB-Q6_K_S`	`Q6_K_S`	⭐⭐⭐⭐⭐ (94-99%)	~3.58 GB	Research, maximum reasoning fidelity
`gemma-4-E2B-it-AB-Q5_K_M`	`Q5_K_M`	⭐⭐⭐⭐ (92-95%)	~3.22 GB	High-end local inference, sweet spot
`gemma-4-E2B-it-AB-Q5_K_S`	`Q5_K_S`	⭐⭐⭐⭐ (90-93%)	~3.04 GB	High-end local inference, sweet spot
`gemma-4-E2B-it-AB-Q4_K_M`	`Q4_K_M`	⭐⭐⭐ (88-92%)	~2.91 GB	Balanced deployment, speed-focused
`gemma-4-E2B-it-AB-Q4_K_S`	`Q4_K_S`	⭐⭐⭐ (87-90%)	~2.79 GB	Ultra-constrained environments

🛠️ Reproducing This Quantization

Exact reproduction is guaranteed using the provided configuration:

Download GGUF Editor D++ v1.0.2.1 - (GGUF.Editor.D++.v1.0.2.1-WinX64.zip)
Load the base gemma-4-E2B-it model
Apply the custom preset: QuantPresets.ini
Run quantization in Impl mode with optimized parameters
Save as GGUF

The QuantPresets.ini file contains all tuned parameters (Rmin, Rdelta, Nstep, UseMAD, per-quant targets) required to replicate this exact weight distribution.

📦 Available Variants

gemma-4-E2B-it-AB-Q6_K_M
gemma-4-E2B-it-AB-Q6_K_S
gemma-4-E2B-it-AB-Q5_K_M
gemma-4-E2B-it-AB-Q5_K_S
gemma-4-E2B-it-AB-Q4_K_M
gemma-4-E2B-it-AB-Q4_K_S

💻 Inference Usage

Compatible with llama.cpp, llama-cpp-python, Ollama, and any GGUF-compatible runtime.

📜 License & Credits

Base Model: Google Gemma 4 E2B-it (License)
Quantization Tool: GGUF Editor D++ by ABBN
Methodology: Custom RMSE/MAD optimization pipeline, per-tensor mapping, v1.0.2 presets

Precision engineering for every bit. Quantized by ABBN.

Downloads last month: 746

GGUF

Model size

5B params

Architecture

gemma4

Hardware compatibility

4-bit

5-bit

6-bit

Model tree for ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor

Base model

google/gemma-4-E2B

Finetuned

google/gemma-4-E2B-it

Quantized

(320)

this model

gemma-4-E2B-it-AB — Precision-Optimized GGUF Quantization

⚙️ Quantization Pipeline & Custom Parameters

📊 Comparative Analysis & Error Metrics

📉 Quantization Fidelity (RMSE/MAE Analysis)

📈 Quality vs. Size Hierarchy

🛠️ Reproducing This Quantization

📦 Available Variants

💻 Inference Usage

📜 License & Credits

Model tree for ABBNDZ/gemma-4-E2B-it-AB-GGUF-Editor

`gemma-4-E2B-it-AB` — Precision-Optimized GGUF Quantization