pearsonkyle/Qwen3.8-27B-GPTQ-W4A16

⚡ Qwen/Qwen3.8-27B · W4A16 · vLLM

GPTQ @ 32K calibration

📦 18.1 GiB · 2.8× smaller compressed-tensors · int4 g128 sym 🧬 calibrated at ctx 32,768 ⚡ MTP bundled · 1.71× decode 👁️ Text + Image · vision 3/3 🏅 tool-calling = bf16, turn for turn 🏗️ vLLM 0.27.1

⚡ What this is

A compressed-tensors W4A16 (GPTQ, int4 group-128 symmetric) quantization of Qwen/Qwen3.8-27B that vLLM serves directly — no conversion, no custom runtime, quantization auto-detected from config.json. Calibrated on 4.26M tokens of real agentic-coding sessions packed at 32K context. Sibling of the GGUF ladder, same corpus: this one for vLLM, the GGUF for llama.cpp / Ollama / LM Studio.

⚡ Speculative decoding + vision, both bundled

The trained MTP draft head ships inside the checkpoint — one --speculative-config flag, no second file, 1.71× decode at 71.2% acceptance. The vision tower ships too, kept at bf16: shown a synthetic test image it named the colour, shape and position of all three shapes correctly, unprompted detail included ("inverted triangle, base horizontal at the top"). Both are additive — drop the flags and you are back to the identical text model.

📐 Read the size honestly5.2 bits per weight overall, not ~4.1. 90.5% of parameters are int4, but embed_tokens and lm_head (1.27 B each, over a 248k vocab) stay bf16 — 4.74 GiB, 26% of the download. Quantizing an output head that wide is the classic rare-token failure mode. The quantized trunk itself is 4.05 bpw.

🛠️ Standard formatPlain compressed-tensors — vLLM serves it directly, no conversion or forks.

🚀 Quick start

Tuned for a 24 GB card; what each flag does is below.

VLLM_ATTENTION_BACKEND=FLASHINFER \
VLLM_FLASHINFER_WORKSPACE_BUFFER_SIZE=201326592 \
vllm serve pearsonkyle/Qwen3.8-27B-GPTQ-W4A16 \
    --max-model-len 106496 \
    --kv-cache-dtype fp8_e4m3 \
    --max-num-seqs 2 \
    --gpu-memory-utilization 0.95 \
    --max-num-batched-tokens 2048 \
    --language-model-only \
    --enable-prefix-caching \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' \
    --enable-auto-tool-choice --tool-call-parser qwen3_xml \
    --reasoning-parser qwen3

--tool-call-parser qwen3_xml is required for tool calling. Qwen3.8 emits XML tool calls (<tool_call><function=NAME>), not JSON. Without it tool_calls is always empty — which looks like a broken quant but is a serving flag.

---

Quality

Measured on vLLM against a bf16 reference served identically — the only valid control. (That same reference scores 0.563 here vs 0.494 on llama.cpp, so these do not compare across stacks.)

	bf16 (ref)	W4A16
Tool-selection accuracy · 174 held-out turns	0.563	0.563
Parameter accuracy	0.345	0.350
Schema-valid rate	0.948	0.948
Malformed tool calls	0	0
Median KLD vs bf16 · 6 distributions	—	0.0021 – 0.0205
Long-context retrieval · 29.7k tokens	—	3/3 codes exact
Vision · colour + shape + position	—	3/3
SWE-rebench dask__dask-11393	—	resolved · 5 steps · 0 malformed

Tool-calling is bf16-identical, turn for turn. KLD is lowest (0.0021) on agentic text — exactly where it was calibrated — and highest (0.0205) on general English.

Read these honestly. SWE-rebench is one instance, not a pass rate. Agentic deltas are noise-bound: vLLM is nondeterministic at temperature=0, and replaying the ladder moved levels by up to 6 turns (3.4pp) — read the endpoints, not the third decimal. Video, multi-image and long-context vision are untested. The vision tower and MTP head are bf16, not quantized (+1.65 GiB), which is why they work.

How it was made — quantization settings, what stayed bf16, calibration corpus

GPTQ W4A16 via llm-compressor 0.13.0 → compressed-tensors 0.18.0: int4, group 128, symmetric, static act-order. Sequential pipeline over all 64 layers (48 linear-attention + 16 full-attention), 496 modules quantized.

Kept bf16 on purpose: lm_head, embed_tokens, vision tower, MTP head. The 248,320-token vocab with untied embeddings makes the two vocab tensors 4.74 GiB, 26% of the download — which is why a "4-bit" 27B lands at 5.2 bpw overall (trunk alone: 4.05) and is bigger than the 14.5 GiB IQ4_XS GGUF. A quantized head over a 248k vocab is the classic rare-token failure mode; the needle test is the check that this worked.

Calibration: 4,255,761 tokens / 3,436 windows at ctx 32,768 — 63% real agentic sessions (CLI logs + SWE trajectories), 6,786 tool calls across 76 schemas, plus reasoning turns, broad-instruct and red-team refusals. GPTQ drew 128 × 32,768-token sequences by deterministic whole-corpus stride.

Built with Quant-Tuner; logs mined with LogMiner.

---

Running it

The Quick start command is tuned for a 24 GB card (RTX 3090): 106,496 tokens of context at ~75 tok/s (45 without MTP), using 23.6 of 24.0 GiB. The download is 18.1 GiB and still fits because only 16 of the 64 layers keep a KV cache — the other 48 are linear attention.

Flag	Why it matters
--tool-call-parser qwen3_xml	Required for tool calls. Qwen3.8 emits XML, not JSON. Without it tool_calls is always empty — which looks like a broken quant but is a serving flag.
--reasoning-parser qwen3	Required for reasoning. Without it the closing </think> is dropped as a special token and reasoning arrives glued onto the answer. With it, reasoning goes to message.reasoning — that name, not reasoning_content.
--kv-cache-dtype fp8_e4m3	Buys the context: halves the cache to 32 KiB/token, worth ~+55K tokens. Needs FlashInfer — on a 3090 the other attention backends refuse 8-bit KV.
--speculative-config … qwen3_5_mtp	The trained MTP draft head ships inside the checkpoint — no second file. ~1.7× decode (75 vs 45 tok/s). Use num_speculative_tokens: 3 on a 3090 (79 vs 74 tok/s); 2 was the optimum on Blackwell (128 tok/s, 1.71×, 71.2% acceptance). Re-tune per card.
--language-model-only	Skips the vision tower, worth ~25K tokens of context. Drop it if you need image input.
--max-num-seqs 2	Each concurrent request costs ~148 MiB of recurrent state on top of KV cache. Raising it to 8 costs ~30K tokens of context.
--gpu-memory-utilization 0.95	Don't raise it. 0.98 measures 131K of context and then dies on the first request.

Reasoning needs a 4096 budget. Reasoning and the answer share max_tokens, and reasoning is spent first — there is no separate thinking budget. A hard problem spent 2,714 tokens thinking; at 1–2K it returns an empty answer with finish_reason: length, which looks like a broken model but is only the budget running out. Set effort per request with {"chat_template_kwargs": {"reasoning_effort": "low"}} (xhigh/high/medium/low), or enable_thinking: false — which measured best for tool calling (0.563 vs 0.437 at xhigh). Tool calls cost only ~22 reasoning tokens either way.

Setup notes — chat template, cold-start caching, FlashInfer

Chat template. Pass --chat-template chat_template_safe_v2.jinja (bundled). The stock template raises on the OpenAI-standard reasoning_effort: "high", so a normal client gets HTTP 400. The safe template fixes that plus three rendering bugs and is byte-identical on 382/382 real holdout prefixes, so adopting it cannot change quality.

The first start after changing any flag reports a smaller KV cache. vLLM compiles the model on a cold cache and still holds that memory while measuring what is free, so you get ~2.2 GiB instead of ~4.3 GiB — often too little to start. The second launch is correct. saved AOT compiled in the log means cold (restart it); Directly load AOT means the numbers are real.

FlashInfer compiles kernels on first use, and a stock pip install vllm cannot: there is no CUDA_HOME (the PyTorch wheels ship a CUDA toolkit but do not put it on PATH), the bundled compiler and CUDA headers can be different versions, and the linker wants lib64, libcudart.so and stubs/libcuda.so, which the wheels do not create. Install ninja, pin nvidia-cuda-nvcc/crt/nvvm to match your PyTorch CUDA version, and symlink those three paths.

Unrelated but expensive: pkill -f "vllm serve" kills your own shell, because the shell's command line contains the pattern too. Use pkill -f "[v]llm serve".

---

License

Apache-2.0, inherited from Qwen/Qwen3.8-27B.