Qwen3.8-27B-GSQ-RCO-IQ3_XXS-Uncensored

A 3-bit GGUF quant of the abliterated Qwen3.8-27B, built to hold full 196K context on a single 16 GB GPU. Sibling size class carries ISTA-DASLab's full-size AIME/GPQA numbers; this file trades recall for residency and says so below.

A 9.73 GiB, mixed-precision GSQ/RCO quant of Huihui's refusal-reduced Qwen3.8-27B, preserving the MTP head and validated for 16 GB GPUs.

This is a quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated, not a new fine-tune. The underlying model is Qwen3.8-27B; the refusal ablation comes from Huihui; the quantization work here follows the GSQ/RCO methodology developed by ISTA-DASLab. The checkpoint is designed to substantially reduce learned refusal behavior. This is not a guarantee of universal compliance — the deployer is responsible for compliant use.

Quick specs

Base	Qwen/Qwen3.8-27B via huihui-ai/Huihui-Qwen3.8-27B-abliterated
File	Qwen3.8-27B-GSQ-RCO-IQ3_XXS-Uncensored-MTP.gguf
Size	10,442,827,584 bytes (9.73 GiB)
Tensors	866 (65 blocks: 64 layers + MTP head)
bpw	3.058 (bytes×8 ÷ 27,320,697,856 params)
MTP head	Present - 15 blk.64.* weights incl. nextn.eh_proj/enorm/hnorm/shared_head_norm, verified non-empty, draft-accept 0.46-0.95 by workload logged against this exact file
Template	froggeric-qwen3.8-tool-use.jinja (local file header v22.5; upstream repo may still show v22.4)
Context	196K resident-fit on 16 GB (q4_0 KV, verified loaded); quality gates ran at 32K server ctx + 64K needle ctx — full-196K retrieval quality pending, not claimed

What I changed

Starting from the abliterated checkpoint, I produced this GGUF using a GSQ/RCO-based non-uniform allocation and IQ3_XXS packing. This isn't a generic IQ3_XXS conversion: sensitive tensors (SSM gates, embeddings, attention gates — 96 tensors) are held at BF16 while the bulk rides the GSQ-coded grid.

The important honesty clause: I reproduced ISTA-DASLab's published per-tensor allocation map for Qwen3.8-27B and applied it to the abliterated weights. I did not independently re-run the multi-GPU RCO budget search. Same map, different base.

I also tried custom importance-matrix blends during the process (fresh + syncretized). They looked promising and then failed needle retrieval outright (0/6 timeouts), so I threw them out and kept the official ISTA imatrix. If a change makes the benchmark prettier but the model worse to actually use, it doesn't ship.

Tested performance (RTX 5070 Ti 16 GB, build2 llama.cpp, this exact file)

Configuration	Decode	Notes
Serial / no spec	~39.6 t/s	400-tok prose, thinking on
MTP n-max 2	~59.3 t/s	+50%. Ship default. Accept 0.46-0.95 by workload (code peaks, prose dips), mean len 1.9-2.9
MTP n-max 2 + --fit off + --spec-draft-backend-sampling + -b 1024	69–82 t/s	Ship flags. The --fit off alone was worth +38%
MTP n-max 3 / 4 (+p-min)	33–37 t/s	Higher acceptance yet slower — verification cost wins. Don't use
KV iq4_nl	26.8 t/s	No fast kernel in this build. Don't use
KV q4_0/q4_0	ship	15.8/16 GB resident

Higher acceptance does not automatically mean higher throughput. The verification cost matters — measure tok/s, not just accept rate.

Suite	Score	Setup
Needle retrieval	6/6	15K-word haystack, depths 0.1–0.9, temp 0.0, 64K server ctx
Toolcall v1 (Pi JSON path)	8/8	tool_call_format=json
Toolcall v2 (args + must-not-fire + chains)	7/14 raw-template; negatives 4/4 clean	Schema path hits a build2 tools+MTP post-boot flake (warm-up rule); Pi-path number pending
Livebench-style	12/12	reasoning/coding/extraction mix
Coherence	4/4	multi-turn continuity

Temp ships at 0.6 (4/8 vs 3/8 raw-template A/B; favored for tool-use reliability in agentic loops). Reasoning: medium default, low for short code, xhigh for verifiable-correctness only.

Academic benchmarks (AIME/GPQA/LiveCodeBench) for this specific file: pending, will append. For reference, ISTA-DASLab's same-bpw base reports AIME25 100.0 / GPQA-Diamond 88.89 / LiveCodeBench v6 84.57 — their base, not this file.

Usage

Froggeric template is required for the intended tool/reasoning behavior (tool_call_format=json for the certified path; template default is xml).

llama-server \
  -m Qwen3.8-27B-GSQ-RCO-IQ3_XXS-Uncensored-MTP.gguf \
  --alias Qwen3.8-27B-GSQ-RCO-IQ3_XXS-Uncensored \
  --jinja --chat-template-file froggeric-qwen3.8-tool-use.jinja \
  --ctx-size 196608 -fa on -ctk q4_0 -ctv q4_0 \
  -ngl 99 -b 1024 -ub 1024 -np 1 -t 8 --fit off \
  --reasoning-budget 256 --spec-type draft-mtp --spec-draft-n-max 2 \
  --spec-draft-backend-sampling

Vision: projector files live in this repo under mmproj/ (BF16 or Q8_0). Start the server with one, then send a multimodal chat request (skip both flags for text-only):

llama-server -m <gguf> --mmproj mmproj/mmproj-Qwen3.8-27B-BF16.gguf --ctx-size 65536 [same flags as above]
# then POST /v1/chat/completions with content blocks including {"type":"image_url",...}

Ollama (text generation supported; tool/reasoning behavior uncertified — use llama-server + Froggeric for that):

FROM ./Qwen3.8-27B-GSQ-RCO-IQ3_XXS-Uncensored-MTP.gguf
PARAMETER num_ctx 196608
PARAMETER num_gpu 999

Honest limitations

Aggressive 3-bit quant — don't expect BF16 behavior. Knowledge recall is weaker than higher-bit variants (the model leans on retrieval/tools instead). MTP output may differ from serial generation on quantized targets; fix the seed and runtime config for reproducibility, disable spec when exact serial behavior matters. Vision + long context gets tight on 16 GB (split text/vision profiles or offload the projector). The imatrix was computed on the stock base, not the abliterated weights — whether that shift matters is untested, not assumed safe.

Files in this repo

File	What
Qwen3.8-27B-GSQ-RCO-IQ3_XXS-Uncensored-MTP.gguf	The quant (SHA256 in SHA256SUMS.txt)
Vision projector	mmproj/mmproj-Qwen3.8-27B-BF16.gguf (0.87 GB) + Q8_0 alt (0.60 GB, 27 tensors fallback-quantized)
froggeric-qwen3.8-tool-use.jinja	Required chat template
REF-IQ3_XXS-mtp.rco-allocation.txt	866-row per-tensor allocation map (authoritative over any summary)
imatrix.dat	ISTA-DASLab's published calibration matrix as used
SHA256SUMS.txt	Hashes
LICENSE	Upstream Apache-2.0 terms apply; see Qwen/Huihui repos

GGUF embeds tokenizer metadata — no separate tokenizer needed for llama.cpp/Ollama/LM Studio/Pi.

Credits

• Qwen — architecture + pretrained weights: Qwen3.8 repo, Qwen3.8-27B
• Huihui — abliterated checkpoint (model), via remove-refusals-with-transformers
• GSQ — Dadgarnia, Tabesh, Nikdan, Helcig, Kurtic, Kleinegger, Alistarh (2026): paper, code
• RCO — Helcig & Alistarh (2026): paper, code
• ISTA-DASLab quant release — Qwen3.8-27B-GSQ-RCO-GGUF (allocation + imatrix source)
• Froggeric — Qwen-Fixed-Chat-Templates
• llama.cpp/GGML — runtime + format; IQ3_XXS is their standard type, nothing custom here

Reproducibility

• Base: huihui-ai/Huihui-Qwen3.8-27B-abliterated
• Allocation: REF-IQ3_XXS-mtp.rco-allocation.txt (in this repo), applied via llama-quantize --tensor-type-file
• Imatrix: imatrix.dat (in this repo), ISTA-DASLab's published matrix as used
• Runtime: den_llama.cpp @ 3231ee89 (build2; -DGGML_CUDA_FA_ALL_QUANTS=ON, CUDA 13.3)
• GPU: RTX 5070 Ti 16 GB, Windows 11, driver per nvidia-smi at run time
• Evals: prompts in evals/prompts/ (hashes in evals/SHA256SUMS), commands in evals/commands.md, raw outputs + timings + accept stats in evals/raw/

Versions

• v1 (current): official imatrix, MTP intact, gates above. Earlier fresh/syncretized-imatrix builds failed needle and were discarded, not shipped.
• v1.1 (in progress): FastMTP-style trained MTP head (4M self-distilled diverse corpus) for deeper-draft acceptance.
• v1.2 (planned): M12 self-distilled corpus (12M agent/tool-heavy trajectories from the v1.1 teacher) + anything else that genuinely moves MTP acceptance at any depth (k1-k4): p-min tuning, draft-KV precision, head-precision ladder. This card updates on release.

Community quant, not affiliated with Qwen, Huihui, or ISTA-DASLab. Bug reports: include model revision, llama.cpp build, ctx/KV/spec settings, sampler. "Feels worse" is fine; reproducible info makes it actionable.