Instructions to use tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS
# Run inference directly in the terminal:
llama cli -hf tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS
# Run inference directly in the terminal:
llama cli -hf tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS
# Run inference directly in the terminal:
./llama-cli -hf tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS
# Run inference directly in the terminal:
./build/bin/llama-cli -hf tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS

Use Docker

docker model run hf.co/tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS

LM Studio
Jan
Ollama
How to use tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF with Ollama:
```
ollama run hf.co/tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS
```
Unsloth Desktop

How to use tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF with Docker Model Runner:
```
docker model run hf.co/tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS
```

Lemonade

How to use tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS

Run and chat with the model

lemonade run user.Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF-IQ3_XXS

List all available models

lemonade list

Hermes Agent

How to use tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "tooltd/Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF:IQ3_XXS" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3.8-27B-IQ4-XS-16GB-VRAM-GGUF / README.md

tooltd

Update README.md

fd54915 verified 3 days ago

preview code

Raw

History Blame Contribute Delete

10.3 kB

metadata

base_model:
  - Qwen/Qwen3.8-27B
license: apache-2.0
tags:
  - unsloth
  - imatrix
  - llama.cpp
  - qwen3.8
  - qwen
  - ubergarm
  - 16GB
  - 12GB

Qwen3.8-27B (GGUF target for 16GB VRAM)

This repository provides GGUF quantizations for Qwen3.8-27B optimized using ZB-ZipBrain, a custom mixed-precision quantization methodology that optimizes LLM tensor bit allocation using rate-distortion marginal cost combined with importance matrix calibration. It automatically identifies Pareto-optimal BPW "sweet spots" to maximize model quality while fitting precise VRAM and memory footprints.
Specifically optimized to fit mainstream GPUs within a 16GB VRAM budget at around 4 BPW, and even 12GB VRAM cards at around 3 BPW.
For filenames marked with v5, I combined ZB + Pelicanmaxxing for visual evaluation, iteratively tuning until the output reached the most stable quality before locking it in.

Benchmark & Evaluation Results

EvalPlus Benchmark Results

HumanEval: 164 tasks
MBPP: 378 tasks
Scores are pass@1, no-thinking mode , kvcache ctk q4_0, ctv q4_0
“—” indicates data not provided.

Quantization	HumanEval	HumanEval+	MBPP	MBPP+	Note
ZB4.00-MIN-v5.1-IQ4_XS	0.945	0.921	0.897	0.780	👑💀
Qwen3.8-27B-IQ4_NL	0.951	0.915	0.902	0.778	bartowski
ZB3.88-MIN-v5-IQ3_M_XL	0.945	0.915	0.91	0.775	—
~~ZB4.00-MIN-v5-IQ4_XS~~	~~0.927~~	~~0.902~~	—	—	~~oldver~~
Qwen3.8-27B-Ridge-3.7bpw	0.933	0.896	0.902	0.765	empero-ai
ZB3.73-MIN-v5.1-IQ3_M_L	0.927	0.896	0.881	0.757	—
~~ZB3.70-MIN-v4-IQ3_M_L~~	~~0.915~~	~~0.896~~	~~0.873~~	~~0.754~~	~~oldver~~
ZB3.00bpw-IQ3_XXS	0.927	0.884	0.865	0.743	—
UD3-IQ4_XS	0.890	0.866	0.897	0.751	unsloth
UD3-Q3_K_XL	0.823	0.805	0.881	0.751	unsloth

Comprehensive Comparison Table

Command llama-perplexity.exe -f /wikitext-2-raw/wiki.test.raw --kl-divergence --kl-divergence-base q38f16baseline.kld -ngl 99 -m model.gguf

All models were evaluated against the BF16 baseline (Mean PPL = 6.950493) using standard Perplexity (PPL) and KL Divergence metrics.

Label	Provider	Size (GB)	Mean KLD	Same Top-p (%)	Mean PPL
UD-Q8_K_XL	unsloth old	29.30	0.000850	98.970%	6.953800
UD-Q6_K_XL	unsloth old	24.14	0.001380	98.520%	6.953600
Q6_K	unsloth old	21.31	0.002290	97.860%	6.950700
Q5_K_M	unsloth old	18.47	0.006220	96.700%	6.974200
UD-Q4_K_XL	unsloth old	16.69	0.008606	96.091%	6.979220
ZB4.97-GOD-IQ4_XS	ZB-GOD	15.82	0.012249	95.337%	7.004243
UD3-Q4_K_S	unsloth UD3	14.30	0.013652	95.149%	6.969514
Autoround-Q4_K_M	Autoround	15.66	0.014657	94.859%	6.950294
ZB4.65-PRO-IQ4_XS	ZB-PRO	14.81	0.015466	94.766%	7.017278
Q4_K_M	unsloth old	15.93	0.015490	94.650%	6.956100
ZB4.60-PRO-IQ4_XS	ZB-PRO	14.65	0.016162	94.668%	7.030895
ZB4.55-PRO-IQ4_XS	ZB-PRO	14.49	0.016647	94.613%	7.032263
IQ4_NL	bartowski	15.20	0.018427	94.230%	7.006472
IQ4_XS	unsloth old	14.63	0.018652	94.270%	7.012695
UD3-IQ4_XS	unsloth UD3	13.27	0.018772	93.975%	7.004732
ZB4.48-STD-IQ4_XS	ZB-STD	14.26	0.018892	94.199%	7.050096
Q4_K_S	unsloth old	15.01	0.018921	94.235%	6.966826
IQ4_XS-i1	mradermacher	14.26	0.019271	94.141%	7.012810
Q4_K_S-i1	mradermacher	14.74	0.019805	93.996%	6.989551
ZB4.36-STD-IQ4_XS	ZB-STD	13.88	0.020556	93.951%	7.054811
ZB4.36-STD-v4-IQ4_XS	ZB-STD	13.88	0.021552	93.886%	6.993211
Q4_0-AutoRound-Code	webhie	14.64	0.026586	92.970%	7.067142
ZB4.14-MIN-IQ4_XS	ZB-MIN	13.19	0.029334	92.799%	7.045689
⭐ZB4.00-MIN-v5.1-IQ4_XS	ZB-MIN	12.74	0.033810	92.309%	7.106519
~~ZB4.00-MIN-v5-IQ4_XS~~	ZB-MIN	12.79	0.034577	92.277%	7.090583
⭐ZB3.88-MIN-v5-IQ3_M_XL	ZB-MIN	12.34	0.042164	91.452%	7.132594
⭐ZB3.73-MIN-v5.1-IQ3_M_L	ZB-MIN	11.88	0.048117	90.759%	7.199937
~~ZB3.70-MIN-v4-IQ3_M_L~~	ZB-MIN	11.82	0.052972	90.363%	7.160063
IQ4_XS-Smaller_3.96	jrell	12.61	0.055499	90.090%	7.252766
Ridge-3.7bpw	empero-ai	11.73	0.118430	85.907%	7.547496
⭐ZB3.0BPW-IQ3_XXS	ZB-MIN	9.62	0.120503	85.320%	7.474669

Update: Aug 20, 2026

The newly released Unsloth Dynamic v3 is truly the best value for performance right now.
My ZB is just an experiment, feel free to check it out for fun :)

Update: Aug 22, 2026

Quant release ZB-4.00 BPW runs cleanly on 16GB VRAM with MTP support and up to 95K context length.

Update: Aug 24, 2026

ZBv3-4.00BPW update maintaining size with KLD and Same top-p performs slightly better. ZBv2-4.00BPW -> ZBv3-4.00BPW: output weights were bumped from Q5_K to Q6_K. Testing shows sharper, more consistent outputs and better one-shot performance. => Go with ZBv3-4.00BPW.
ZB3.7-MIN ⚔️ empero-ai/Qwen3.8-27B-Ridge. 🤣
I have just updated empero-ai/Qwen3.8-27B-Ridge. benchmarks; comparing my ZB 3.7bpw metrics, it looks like it easily beats down Qwen3.8-27B-Ridge

Update: Aug 26, 2026

ZB3.7-MIN-v4 update PPL slightly better

Update: Aug 28, 2026

All use Q6_K for output.weight
Please prioritize later development versions, I have removed older ones because I was not satisfied with them.
Quant release 4.00bpw & 3.88bpw v5 (v5 = ZB + Pelicanmaxxing and visually check for other aspects of stability. )
ZB3.88-MIN-v5-IQ3_M_XL ⚔️ UD3-Q3_K_XL 😎 If anyone has used this pair, please let me know what you think.
3.0BPW-IQ3_XXS: New release size only 9.62 GB 🙀 with metric comparable to Ridge—3.7 bpw.

Update: Sep 5, 2026

ZB4.00-MIN-v5.1-IQ4_XS This version has been updated so that all tensors are ≥ IQ3_XXS , previous version contained some IQ2_S tensors. Quality is slightly improved, new file size saves 50MB.

Update: Sep 8, 2026

ZB3.73-MIN-v5.1-IQ3_M_L Added new 3.73 bpw. Updated all tensors ≥ IQ3_XXS, quality is slightly improved. Change for ZB3.70-MIN-v4-IQ3_M_L.
Added HumanEval, MBPP benchmark

Recommended Settings: Set reasoning_effort to medium. At this BPW level, it delivers much more stable outputs and fits well in agentic workflows. You can also use the default settings for higher quality, though it will take longer.

llama-server -m models/qwen38/Qwen3.8-27B-ZB4.00-MIN-IQ4_XS.gguf -mm models/qwen38/Qwen3.8-27B-mmproj-BF16.gguf --host 0.0.0.0 --port 8080 --temp 1 --top-p 0.95 --top-k 20 --min-p 0.00 --reasoning-preserve -ctk q4_0 -ctv q4_0 -fa on --ubatch-size 384 --batch-size 384 --no-mmproj-offload --spec-type draft-mtp,ngram-mod --spec-draft-n-max 2 --spec-ngram-mod-n-match 24 --spec-ngram-mod-n-min 24 --spec-ngram-mod-n-max 32 -ngl 99 -t 7 --ctx-size 95000 -np 1 --load-mode mlock --image-min-tokens 1024 --image-max-tokens 2048 --chat-template-kwargs '{\"reasoning_effort\": \"medium\"}'

ZB Tiers & Recommendations

ZB-GOD : God. A singularity appears. Reaches 0.012249 Mean KLD and 95.34% top-probability match.
ZB-PRO : Pro. For 16 GB VRAM GPUs with offloading on CPU. Balances quality output with substantial size savings.
ZB-STD : Standard. Similar to other standard IQ4_XS models currently available.
ZB-MIN : Minimal. Optimal footprint for tight 16 GB memory setups, allowing headroom for longer context windows

Credits & Acknowledgements

Base Model: Qwen3.8 27B by Alibaba Cloud / Qwen Team.
BF16 Base GGUF: Provided by Unsloth AI.
Importance Matrix (imatrix): Generated and curated by ubergarm.
Inference & Quantization Framework: llama.cpp by Georgi Gerganov and contributors.