Instructions to use anlord/gemma-3-1b-it-Abliterated-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use anlord/gemma-3-1b-it-Abliterated-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf anlord/gemma-3-1b-it-Abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf anlord/gemma-3-1b-it-Abliterated-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf anlord/gemma-3-1b-it-Abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf anlord/gemma-3-1b-it-Abliterated-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf anlord/gemma-3-1b-it-Abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf anlord/gemma-3-1b-it-Abliterated-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf anlord/gemma-3-1b-it-Abliterated-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf anlord/gemma-3-1b-it-Abliterated-GGUF:Q4_K_M

Use Docker

docker model run hf.co/anlord/gemma-3-1b-it-Abliterated-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use anlord/gemma-3-1b-it-Abliterated-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "anlord/gemma-3-1b-it-Abliterated-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "anlord/gemma-3-1b-it-Abliterated-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/anlord/gemma-3-1b-it-Abliterated-GGUF:Q4_K_M

Ollama
How to use anlord/gemma-3-1b-it-Abliterated-GGUF with Ollama:
```
ollama run hf.co/anlord/gemma-3-1b-it-Abliterated-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use anlord/gemma-3-1b-it-Abliterated-GGUF with Docker Model Runner:
```
docker model run hf.co/anlord/gemma-3-1b-it-Abliterated-GGUF:Q4_K_M
```

Lemonade

How to use anlord/gemma-3-1b-it-Abliterated-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull anlord/gemma-3-1b-it-Abliterated-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.gemma-3-1b-it-Abliterated-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

gemma-3-1b-it-Abliterated-GGUF

GGUF quantizations of gemma-3-1b-it-Abliterated.

The base model was abliterated using AnlordAbliterator 1.4.0 and then converted to GGUF and quantized into multiple formats.

Available Quantizations

Quantization	File
BF16	`gemma-3-1b-it-abliterated-bf16.gguf`
F16	`gemma-3-1b-it-abliterated-f16.gguf`
Q8_0	`gemma-3-1b-it-abliterated-q8_0.gguf`
Q6_K	`gemma-3-1b-it-abliterated-q6_k.gguf`
Q5_K_M	`gemma-3-1b-it-abliterated-q5_k_m.gguf`
Q5_0	`gemma-3-1b-it-abliterated-q5_0.gguf`
Q4_K_M	`gemma-3-1b-it-abliterated-q4_k_m.gguf`
Q4_0	`gemma-3-1b-it-abliterated-q4_0.gguf`

Which Quantization Should I Use?

A simple rule of thumb:

Quantization	Quality	Size	Recommended for
BF16	★★★★★	Very large	Maximum precision
F16	★★★★★	Large	Maximum precision
Q8_0	★★★★★	Large	Near-original quality
Q6_K	★★★★★	Medium	High quality
Q5_K_M	★★★★☆	Medium	Quality / size balance
Q5_0	★★★★☆	Medium	General use
Q4_K_M	★★★★☆	Small	Recommended default
Q4_0	★★★☆☆	Smallest	Maximum memory savings

Q4_K_M is the recommended starting point for most users who want a good balance between quality and memory usage.

Base Model

google/gemma-3-1b-it

Original model:

https://huggingface.co/google/gemma-3-1b-it

Abliterated Transformers version:

https://huggingface.co/anlord/gemma-3-1b-it-Abliterated

Abliteration

The base model was processed with AnlordAbliterator 1.4.0.

Results

Model: google/gemma-3-1b-it

Initial refusals: 97 / 104
Final refusals:    5 / 104

KL divergence: 0.10103859007358551

200 optimization trials

Tool

AnlordAbliterator

Running with llama.cpp

Example:

llama-cli -m gemma-3-1b-it-abliterated-q4_k_m.gguf

The GGUF files are intended for use with GGUF-compatible software such as llama.cpp and other compatible inference applications.

License

This repository contains derivative model files based on google/gemma-3-1b-it.

The original gemma-3-1b-it model is released under the Gemma Terms of Use and is gated on Hugging Face.

Refer to the original model repository for the applicable license terms.

Disclaimer

These quantizations are derived from an abliterated version of gemma-3-1b-it.

Quantization may introduce small differences in model behavior and output quality compared with the original Safetensors model.

Downloads last month: 454

GGUF

Model size

1.0B params

Architecture

gemma3

Hardware compatibility

4-bit

5-bit

6-bit

8-bit

16-bit

Model tree for anlord/gemma-3-1b-it-Abliterated-GGUF

Base model

google/gemma-3-1b-pt

Finetuned

google/gemma-3-1b-it

Quantized

(477)

this model