lued/Qwen3.8-27B-huihui-abliterated-INT8-W8A16-DFlash2

Huihui-Qwen3.8-27B-abliterated · INT8 W8A16 · DFlash2

Fast, near-lossless Huihui abliterated Qwen3.8-27B for dual RTX 3090s.

Base model · Qwen3.8-27B · vLLM · llm-compressor

A numerical INT8 W8A16 quantization of huihui-ai/Huihui-Qwen3.8-27B-abliterated, an abliterated fine-tune of Qwen/Qwen3.8-27B, served with the DFlash2 drafter for speculative decoding. All model credit belongs to huihui-ai and Qwen; this repository changes numerics only. The detailed engineering notes live in TECHNICAL.md.

The short version

Huihui-Qwen3.8-27B-abliterated is a big model. This build makes it fast on consumer hardware:

• Up to ~117 t/s decode on two RTX 3090s at full 262K context (family baseline)
• ~1.4x to 1.5x faster than the same model with native MTP4 speculation (family baseline, pending own bench)
• ~2.2x faster than plain autoregressive decoding (family baseline, pending own bench)
• 98.5% top-1 agreement with the abliterated BF16 model (measured: mean KLD 0.000694)
• 28 GiB on disk (6 shards), 2.02 GiB drafter, full 262K context on 2×24 GB

The KLD and audit numbers above are this build's own measured evidence. Decode throughput figures are the family baseline from the identical sibling build; this build's own serve run is pending (see Status).

How fast

Measured on this build: KLD 0.000694 nats mean, 98.5% top-1. Decode numbers below are the family baseline (single stream, same engine, same 262K context, cold cache). Generation tokens per second.

Prompt tokens	Autoregressive	MTP4	DFlash2	DFlash2 vs MTP4
128	47	77	117	1.5x
2,048	47	73	103	1.4x
8,192	47	75	102	1.4x

DFlash2 proposes 7 draft tokens per step and the target verifies them all in one pass. Native MTP proposes 4. More drafts per verification step means more accepted tokens and less idle GPU time.

How close to the original

	BF16 original	This build
Size	~56 GiB	28 GiB (6 shards)
Mean KLD	0	0.000694
Top-1 agreement	100%	98.5%

KLD is a fancy way of asking "does the quantized model pick the same next token as the original?" Lower is closer. A mean of 0.000694 nats means the INT8 weights and the BF16 weights are nearly interchangeable. Halving the size is what lets the full 262K context fit on two 3090s.

What you get

• Full 262,144-token context on 2×RTX 3090 (intended, pending serve)
• Vision tower, thinking controls, and tool calling intact, same as upstream huihui-ai/Huihui-Qwen3.8-27B-abliterated
• Native MTP removed: DFlash2 replaces it, so there is no dead weight in the repo
• Runs on vLLM with the club-3090 patch set (DFlash2 is a new spec-decoder, still in PR review upstream)

Run it

Three steps. The full command with every flag is in TECHNICAL.md.

# 1. Models (pulls into ~/.cache/huggingface)
hf download lued/Qwen3.8-27B-huihui-abliterated-INT8-W8A16-DFlash2
hf download lued/Qwen3.8-27B-DFlash2-W8

# 2. Patches (DFlash2 support is not in a released vLLM yet)
git clone https://github.com/noonghunna/club-3090.git

# 3. Serve (condensed; full command in TECHNICAL.md)
export CLUB3090="$HOME/club-3090"
podman run --rm --replace --device nvidia.com/gpu=all --ipc=host -p 8080:8080 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v "$CLUB3090/models/qwen3.6-27b/vllm/patches/vllm-pr52816-dflash2":/etc/club3090/pr52816:ro \
  -v "$CLUB3090/models/qwen3.6-27b/vllm/patches/vllm-pr48375-mamba-drop-eagle-block":/etc/club3090/pr48375:ro \
  --entrypoint bash docker.io/vllm/vllm-openai:nightly-5a4c8d99242e9e069b604d0e9b969e77f7dd501d \
  -c 'bash /etc/club3090/pr48375/install.sh || exit 1; bash /etc/club3090/pr52816/install.sh || exit 1; exec vllm serve "$@"' -- \
  lued/Qwen3.8-27B-huihui-abliterated-INT8-W8A16-DFlash2 \
  --tensor-parallel-size 2 --max-model-len 262144 --kv-cache-dtype fp8_e4m3 \
  --speculative-config '{"method":"dflash","model":"lued/Qwen3.8-27B-DFlash2-W8","num_speculative_tokens":7}'

Status

Built, audited, and published 2026-08-20 to lued/Qwen3.8-27B-huihui-abliterated-INT8-W8A16-DFlash2 at commit 26c9c9bc. STRUCTURAL AUDIT PASS (401 packed / 783 preserved, independently re-verified tensor-by-tensor), DFLASH2 AUDIT PASS (15 MTP stripped, embed packed), shared drafter audit PASS. KLD measured: mean 0.000694 nats, top-1 0.9850, sanity gate PASS. Registered in the HF cache and verified to resolve offline. Serve-validation on this rig is pending (family baseline decode from the identical sibling build). The drafter ships separately as lued/Qwen3.8-27B-DFlash2-W8. All engineering detail, audit evidence, and measured tables are in TECHNICAL.md.