Swift-1.5-Qwen3.8-27B-Uncensored-GPTQ-Int4-sym-G128-MTP-BF16

GPTQ-Int4 quantization of ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-MTP, built for Intel Arc / vLLM XPU with the MTP head preserved in BF16 so native speculative decoding still works.

The recipe is a reproduction of kernelogic/Qwen3.8-27B-Uncensored-GPTQ-Int4-sym-G128-MTP-BF16, applied to the Swift lineage instead of the JonathanColetti one. Same quantizer version, same config, same MTP handling.

Why this exists

Quantizing this model naively breaks speculative decoding. The 15 mtp.* tensors are the draft head; if they get quantized along with everything else, draft acceptance collapses and you lose roughly half your decode speed. The fix is one line of quantize config:

dynamic={"-:.*mtp.*": {}}   # exclude mtp.* from quantization -> stays BF16

The result has 400 quantized weight tensors (I32) + 15 preserved BF16 MTP tensors.

There is a second, less obvious requirement. transformers' Qwen3_5 declares _keys_to_ignore_on_load_unexpected = [r"^mtp.*"], so the draft tensors are dropped at load and never become modules — the dynamic exclusion on its own emits a checkpoint with no draft head at all, silently. What writes them back is gptqmodel's out_of_model_tensors = {"prefixes": ["mtp"]}, which re-reads them from the source safetensors at save time. That is also why they stay BF16 while the body is fp16: they are copied bytes, never converted.

Provenance

Qwen/Qwen3.8-27B                          (base)
  └─ ukisai/Swift-1.5-Qwen3.8-27b                 (Swift 1.5, reasoning-efficient finetune)
       └─ ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-MTP
            (uncensored, orcarouter rank-1 refusal direction projected out, BF16)
              └─ this repo                             (GPTQ-Int4, 20 GB)

Quantization

Tool	gptqmodel 7.3.2
Config	bits=4, group_size=128, sym=True, desc_act=False, lm_head=False, pack_dtype=int32
MTP	dynamic={"-:.*mtp.*": {}} → 15 tensors kept BF16
Calibration	256 samples @ 2048 tokens from allenai/c4 (general web text)
Time	21.1 min of per-module quantization across 64 layers (sum of quant_log.csv; excludes calibration forwards, offload and packing, so wall clock is higher)
Output	5 shards, 20 GB, 2399 tensors

Group size is 128, verified from the tensor shapes (qweight.rows * 8 / scales.rows) rather than taken from the config's own declaration. lm_head and the vision tower (333 tensors) are left unquantized.

Measured performance

Not yet measured on this build. The structural properties above are read from the checkpoint; throughput and MTP acceptance are not derivable from one and have to be measured on the target hardware.

For reference, the checkpoint this recipe reproduces reports 68.5 tok/s on code generation and 52.4 on prose (single Arc Pro B70, 230 W, MTP4, fp8 KV, 131,072 context) — that is a different checkpoint on a different serving stack, so treat it as an order of magnitude, not a prediction for this one.

Throughput on this architecture tracks MTP acceptance, which depends on how predictable the output is: code drafts well, freeform prose less so. Any number quoted without an acceptance figure alongside it is not reproducible.

Serving (vLLM XPU)

vllm serve /model \
  --quantization gptq --dtype float16 \
  --kv-cache-dtype fp8 \
  --max-num-batched-tokens 8192 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":5}' \
  --chat-template /model/chat_template.jinja

Qwen3.8 MTP on XPU needs a patched vLLM. The build used to produce and serve this checkpoint is at github.com/greglechin/vllm-xpu; the Intel Arc Pro B70 cookbook is the other published route. Context length, --tensor-parallel-size and --gpu-memory-utilization are deliberately omitted above because they depend on your card count and VRAM — size them from your own boot log.

Three things that will bite you

1. MTP depth is not free, and tok/s is the wrong way to judge it. Read the per-position acceptance curve from the engine's SpecDecoding metrics, not the run mean: a decay ratio around 0.8 pays for the extra drafter pass and around 0.68 is break-even. Changing depth typically moves throughput by less than boot-to-boot variance, so a single A/B run measures nothing.
2. Prefix caching depends on your engine build. Qwen3.8 is a hybrid GDN/Mamba architecture and upstream vLLM does not declare supports_mamba_prefix_caching for it, which is why the reference checkpoint advises disabling it (measured there: 3,036 queries → 0 hits). With the mamba-align patches and --mamba-cache-mode align it does work — 84.8–85.3% hit rate measured on the serving stack above. Check your own hit-rate metric rather than assuming either outcome.
3. --kv-cache-dtype fp8 is effectively required on 32 GB cards at long context; fp16 KV does not fit.

Limitations — please read

• No quality evaluations were run on this quantization. No perplexity comparison against the BF16 source, no coding or reasoning benchmarks, no quantitative refusal-rate testing. What was verified is structural: tensor counts and dtypes, the preserved BF16 MTP head, complete quantized-module tensor sets, group size derived from shapes, and an unquantized lm_head and vision tower. If you need quality guarantees, measure before relying on it.
• Calibration used general web text with no code. If coding quality matters to you, a code-inclusive calibration set would likely be better. This is inherited from the reference recipe, which names the same limitation.
• Uncensoring is inherited, not verified here. All refusal-removal properties come from the upstream ablation (ajgazin/Swift-1.5-Qwen3.8-27B-Uncensored-MTP publishes its own refusal evaluation); this repo only changes numeric precision. Quantization was not evaluated for its effect on that behaviour in either direction.
• The upstream publishes a KL divergence and states what it did not measure. It reports KL 0.0884 against ukisai/Swift-1.5-Qwen3.8-27b on first-token distributions over 100 harmless prompts, and 23/100 refusals, both under Heretic's built-in evaluation with the reference rows measured the same way. Its refusal direction is recovered from orcarouter's published weights rather than fitted fresh, and it verifies that recovery by reproducing orcarouter's 131 edited tensors to 99.75% bit-identical. What it explicitly does not evaluate: general benchmarks, refusal behaviour in thinking mode, whether Swift 1.5's shorter reasoning traces survive, and MTP acceptance — which on a speculative-decoding deployment is the number that decides decode throughput. Measure acceptance on your own traffic before promoting this over a checkpoint you have already gated.
• This is an uncensored model. It will attempt requests an aligned model declines. You are responsible for how you use it.

Licence

Not Apache-2.0. These weights inherit the Swift Open License v1.0 from ukisai/Swift-1.5-Qwen3.8-27b. Personal, research, educational, evaluation and commercial use are free for individuals and organizations with annual recurring revenue, including affiliates, of up to US$1,000,000. Above that threshold, commercial use requires a separate Swift Enterprise License — contact UkisAI.

Credits

• Qwen — base model
• UkisAI — Swift finetune
• ajgazin — directional ablation
• kernelogic/Qwen3.8-27B-Uncensored-GPTQ-Int4-sym-G128-MTP-BF16 — the quantization recipe this reproduces
• gptqmodel — quantization