sakamakismile/gpt-oss-20b-abliterated-EXL3-6bpw

gpt-oss-20b-abliterated — EXL3 6.0 bpw

An ExLlamaV3 EXL3 6.0 bpw quantization of wangzhang/gpt-oss-20b-abliterated (a refusal-suppressed openai/gpt-oss-20b, 21B total / 3.6B active, 32 experts). Baked and measured by Lna-Lab (YUKI) on 2026-09-03. Sibling: gpt-oss-120b-abliterated-EXL3-6bpw.

Size	17 GB — does not fit one 16 GB card; use two GPUs or -mcl (experts on CPU)
Bits	6.0 bpw decoder, 8 bpw head (-hb 8 -hq -cb mul1)
Context	128k (native)
Format	harmony — use the bundled chat_template.jinja

Why this exists

Stock ExLlamaV3 1.4.5 mis-loads bf16 HF-layout gpt-oss experts (gate/up interleave ignored, expert biases dropped) and the quantized model speaks noise. This repo was baked with the fix described in the 120B card and verified (layer-0 MoE vs transformers reference: relative error 4e-4). OpenAI's MXFP4 checkpoints are unaffected.

Measured speed (RTX PRO 2000 Blackwell 16 GB, ExLlamaV3 1.4.5, 128 tokens, no speculation)

Single stream:

Form	ja	en	code	VRAM
TP=2 (2 × 16 GB)	63	75.2	— tok/s	~8.5 GB / GPU
layer-split, 2 GPUs	50.4	50.2	50.1 tok/s	15.2 + 2.2 GB
1 GPU + experts on CPU (-mcl 24)	41.4	44.8	42.0 tok/s	2.4 GB

Multi-stream (en, aggregate / per stream): TP=2 — 1: 75 / 75, 2: 107 / 53, 4: 121 / 30, 8: 139 / 17; layer-split — 1: 49, 2: 72, 4: 83, 8: 89. Monotonic (unlike the 120B on TP=7).

Prefill (cold, tokens/s at 256 / 1k / 4k / 16k): 2 GPUs 1,169 / 1,844 / 3,249 / 3,541 (16k in 4.6 s); 1 GPU + CPU experts 562 / 1,257 / 2,512 / 2,861 (16k in 5.7 s).

Running it

pip install exllamav3==1.4.5     # plus the SlidingAttention patch described in the 120B card
CUDA_VISIBLE_DEVICES=0,1 python3 lna-lab/serve-gptoss.py -m . -tp --port 8020        # 2 GPUs, tensor parallel
CUDA_VISIBLE_DEVICES=0   python3 lna-lab/serve-gptoss.py -m . -mcl 24 --port 8020    # 1 GPU + CPU experts

lna-lab/serve-gptoss.py is a small OpenAI-compatible harmony server (analysis → reasoning_content, commentary tool calls → tool_calls, final → content, SSE with usage, reasoning_effort, sampling knobs, structured tool-failure results). See the 120B card for the ExLlamaV3 1.4.5 patches and the jinja2 tojson trap the server already works around.

Files

• model-*.safetensors, config.json, tokenizer.json, chat_template.jinja, generation_config.json
• LNA-LAB-NOTES.md — recipe, traps and measurements (Japanese)
• lna-lab/ — server, meters, bake script, upstream model card

Credits

• Base model and abliteration: wangzhang (abliterix)
• Original weights: OpenAI, Apache-2.0
• ExLlamaV3: turboderp
• Bake, measurements, server: Lna-Lab / YUKI (@Tono_Ken3)