Swift-Qwen3.8-27B-W4A16-syv-fast / README.md

Upload README.md with huggingface_hub

5e9ab3f verified 14 days ago

3.86 kB

license: other
license_name: swift-open-license-1.0
license_link: LICENSE
library_name: vllm
base_model:
  - ukisai/Swift-Qwen3.8-27b
  - TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ
tags:
  - qwen
  - qwen3.8
  - swift
  - w4a16
  - gptq
  - compressed-tensors
  - speculative-decoding
  - mtp
  - rtx-3090
  - syv
pipeline_tag: text-generation

Swift-Qwen3.8-27B-W4A16-syv-fast

The syv-ai/qwen38-27b-rtx3090 fast-variant serving shape of ukisai/Swift-Qwen3.8-27b (the "reduced reasoning" finetune of Qwen3.8-27B), built entirely from Swift's own weights and outputs:

Body: TheUnderscore's W4A16 asymmetric-AWQ g128 compressed-tensors quant of Swift, carried through unmodified.
lm_head + MTP: int4 GPTQ (g128, symmetric) calibrated on Swift's own hidden states — 300k rows captured through the syv drafter/ pipeline. lm_head KL to the bf16 head: 0.00234 (RTN: 0.00707; the official syv Qwen fast variant shipped at 0.0029). MTP relative errors 0.146–0.176, inside the range of the shipped syv fast variant.
Embeddings: int8 g128, requantized from Swift's own weights.
Draft vocab: counted over 4.23M tokens of Swift's own outputs on a coding-agent-weighted prompt corpus (35% Rust, 16% TypeScript, 11% debugging, 10% code-edit, 8% agent/tool-use, 10% architecture, 7% technical reasoning, 3% general; 61% thinking-on). Swift emits far fewer distinct tokens than base Qwen (~25.9k vs ~54k), so the draft head is 25,879 rows, not the base model's 40,960.

Held-out coverage (10% of sequences, never counted against):

draft vocab	all tokens	code sources
Swift-derived (this model)	99.81%	99.86%
base-Qwen 40k list	96.69%	96.67%

Serving

Designed for the syv single-user stack (vLLM 0.28.0 + the repo's patch series — the MTP draft-vocab patch is required for the 25,879-row head; stock vLLM will not use it):

MODEL=/path/to/Swift-Qwen3.8-27B-W4A16-syv-fast \
  SPEC=mtp PREFIX_CACHE=1 CTX=long MAX_LEN=114688 \
  bash single-user/start_qwen.sh    # from the syv checkout

The syv verify.sh will report one false FAIL on this dir (it asserts an int8 lm_head; this is int4 by construction, like the official fast variant).

Measured (RTX 3090, syv stack, MTP + prefix caching, 114,688 context)

	decode tok/s	MTP acceptance	tok/step
this model	98.4	0.660	2.98
Swift + int8 heads, base draft vocab	94.0	0.630	2.89
official syv Qwen fast variant	98.2	0.634	2.90

Quality battery (identical prompts): 8/9, matching the int8 build — tool calling, strict JSON, streaming and the qwen3 reasoning parser all clean. Swift's reasoning-termination behaviour is preserved (GPTQ heads verified not to shift it, including under greedy decoding).

Provenance & licence

Attribution chain: Alibaba Cloud Qwen3.8-27B (Apache-2.0, included as LICENSE-APACHE-2.0) → UkisAI Swift finetune (Swift Open License v1.0, included as LICENSE) → TheUnderscore W4A16-AWQ body (same licence) → this repository's int4-GPTQ heads, int8 embeddings and Swift-derived draft vocabulary (quantisation and calibration by liamwh, using the syv-ai/qwen38-27b-rtx3090 drafter/ pipeline with a coding-agent-weighted Swift corpus). Distributed under the Swift Open License v1.0; commercial use above its revenue threshold requires the Swift Enterprise License.

Rebuild from scratch with the syv checkout's drafter/ pipeline (gen_data.py → capture.py → GPTQ heads → build_draft_vocab.py) over a Swift-weighted prompt corpus; swift_draft_vocab_ids.json here is the exact id list this model serves.