Upload README.md with huggingface_hub

README.md

@@ -1,75 +1,96 @@
 ---
 license: other
 license_name: swift-open-license-1.0
-library_name: transformers
-pipeline_tag: image-text-to-text
-base_model: ukisai/Swift-Qwen3.8-27b
+license_link: LICENSE
+library_name: vllm
+base_model:
+  - ukisai/Swift-Qwen3.8-27b
+  - TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ
 tags:
   - qwen
-  - qwen3.5
   - qwen3.8
-  - finetune
-  - awq
-  - int4
+  - swift
   - w4a16
+  - gptq
   - compressed-tensors
-  - lmdeploy
+  - speculative-decoding
+  - mtp
+  - rtx-3090
+  - syv
+pipeline_tag: text-generation
 ---
 
-# Swift-Qwen3.8-27B-W4A16-AWQ
-
-W4A16 (4-bit weights, 16-bit activations) AWQ compressed-tensors quantization
-of [`ukisai/Swift-Qwen3.8-27b`](https://huggingface.co/ukisai/Swift-Qwen3.8-27b).
-
-## Quantization method
-
-- **Scheme:** `W4A16_ASYM` — 4-bit asymmetric per-group quantization (group size 128) of all `Linear` weights, stored in the compressed-tensors pack-quantized format (`weight_packed` / `weight_scale` / `weight_zero_point` / `weight_shape`), which LMDeploy `turbomind` auto-detects and loads natively (including the MTP head and vision tower, which stay BF16).
-- **Tooling:** [llmcompressor](https://github.com/vllm-project/llmcompressor) one-shot offline quantization with CPU offloading (`compressed_tensors.offload.load_offloaded_model`), so the full-precision source fits on a 2×16 GB VRAM setup.
-- **AWQ activation smoothing:** `AWQModifier` with the layer-scoped hybrid-attention mappings from `build_hybrid_attention_mappings` — full-attention `input_layernorm` → `self_attn.q/k/v`, `post_attention_layernorm` → `mlp.gate/up`, and `mlp.up_proj` → `mlp.down_proj`, with `duo_scaling="both"` and CPU offload, followed by W4A16 quantization. This layer-scoped recipe is required for hybrid-attention (Qwen3.5-family) architectures — grouped-regex smoothing or mismatched mappings corrupt decoding.
-- Unquantized (kept BF16): embeddings, `lm_head`, norms, `linear_attn.in_proj_a/b`, the vision tower, and the MTP head.
-- **Run command:**
+# Swift-Qwen3.8-27B-W4A16-syv-fast
+
+The [syv-ai/qwen38-27b-rtx3090](https://github.com/syv-ai/qwen38-27b-rtx3090)
+**fast-variant serving shape** of
+[ukisai/Swift-Qwen3.8-27b](https://huggingface.co/ukisai/Swift-Qwen3.8-27b)
+(the "reduced reasoning" finetune of Qwen3.8-27B), built entirely from
+Swift's own weights and outputs:
+
+- **Body**: TheUnderscore's W4A16 asymmetric-AWQ g128 compressed-tensors
+  quant of Swift, carried through unmodified.
+- **lm_head + MTP**: int4 GPTQ (g128, symmetric) calibrated on **Swift's own
+  hidden states** — 300k rows captured through the syv `drafter/` pipeline.
+  lm_head KL to the bf16 head: **0.00234** (RTN: 0.00707; the official syv
+  Qwen fast variant shipped at 0.0029). MTP relative errors 0.146–0.176,
+  inside the range of the shipped syv fast variant.
+- **Embeddings**: int8 g128, requantized from Swift's own weights.
+- **Draft vocab**: counted over 4.23M tokens of **Swift's own outputs** on a
+  coding-agent-weighted prompt corpus (35% Rust, 16% TypeScript, 11%
+  debugging, 10% code-edit, 8% agent/tool-use, 10% architecture, 7%
+  technical reasoning, 3% general; 61% thinking-on). Swift emits far fewer
+  distinct tokens than base Qwen (~25.9k vs ~54k), so the draft head is
+  25,879 rows, not the base model's 40,960.
+
+Held-out coverage (10% of sequences, never counted against):
+
+| draft vocab | all tokens | code sources |
+|---|---|---|
+| **Swift-derived (this model)** | **99.81%** | **99.86%** |
+| base-Qwen 40k list | 96.69% | 96.67% |
+
+## Serving
+
+Designed for the syv single-user stack (vLLM 0.28.0 + the repo's patch
+series — the MTP draft-vocab patch is required for the 25,879-row head;
+stock vLLM will not use it):
 
 ```bash
-CUDA_VISIBLE_DEVICES=1,2 python3 quantize-awq-hybrid.py \
-  --model_path ./Swift-Qwen3.8-27b \
-  --quant_path ./Swift-Qwen3.8-27b-W4A16-AWQ \
-  --offload_dir ./Swift-Qwen3.8-27b-W4A16-AWQ
+MODEL=/path/to/Swift-Qwen3.8-27B-W4A16-syv-fast \
+  SPEC=mtp PREFIX_CACHE=1 CTX=long MAX_LEN=114688 \
+  bash single-user/start_qwen.sh    # from the syv checkout
 ```
 
-(Default calibration: UltraChat 200k `train_sft`.)
-
-## Recommended parameters
-
-From the [base model](https://huggingface.co/ukisai/Swift-Qwen3.8-27b): temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 0, repetition_penalty 1.0.
-
-## Usage
-
-Tested with LMDeploy `turbomind`:
-
-```bash
-from lmdeploy import pipeline, TurbomindEngineConfig
-
-pipe = pipeline(
-    "TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ",
-    backend_config=TurbomindEngineConfig(
-        tp=2,
-        model_format="compressed-tensors",
-        language_model_only=True,
-    ),
-)
-print(pipe("Hello, who are you?").text)
-```
-
-## License and access
-
-Swift weights are distributed through gated access under the **Swift Open License v1.0**;
-this quantization inherits that license from the base model. Personal, research,
-educational, evaluation, and commercial use are free for individuals and organizations
-with annual recurring revenue, including affiliates, of up to US$1,000,000. Above that
-threshold, commercial use requires a separate **Swift Enterprise License**. Contact
-[UkisAI](https://ukisai.com/contact) for terms.
-
-## Files
-
-- `quantize-awq-hybrid.py` — the script used to produce this quantization (CPU-offloaded, DDP/torchrun, produces properly numbered `-of-N` shards). `--offload_dir` selects where per-rank CPU offload temp folders live (defaults to the current working directory).
-- `model-nonquant.safetensors` — unquantized tensors (`mtp.*` and `model.visual.*`) preserved BF16 so the full model architecture is loadable.
+The syv `verify.sh` will report one false FAIL on this dir (it asserts an
+int8 lm_head; this is int4 by construction, like the official fast variant).
+
+## Measured (RTX 3090, syv stack, MTP + prefix caching, 114,688 context)
+
+| | decode tok/s | MTP acceptance | tok/step |
+|---|---|---|---|
+| **this model** | 98.4 | **0.660** | 2.98 |
+| Swift + int8 heads, base draft vocab | 94.0 | 0.630 | 2.89 |
+| official syv Qwen fast variant | 98.2 | 0.634 | 2.90 |
+
+Quality battery (identical prompts): 8/9, matching the int8 build — tool
+calling, strict JSON, streaming and the qwen3 reasoning parser all clean.
+Swift's reasoning-termination behaviour is preserved (GPTQ heads verified
+not to shift it, including under greedy decoding).
+
+## Provenance & licence
+
+Attribution chain: **Alibaba Cloud** Qwen3.8-27B (Apache-2.0, included as
+`LICENSE-APACHE-2.0`) → **UkisAI** Swift finetune (Swift Open License v1.0,
+included as `LICENSE`) → **TheUnderscore** W4A16-AWQ body (same licence) →
+this repository's int4-GPTQ heads, int8 embeddings and Swift-derived draft
+vocabulary (quantisation and calibration by **liamwh**, using the
+[syv-ai/qwen38-27b-rtx3090](https://github.com/syv-ai/qwen38-27b-rtx3090)
+`drafter/` pipeline with a coding-agent-weighted Swift corpus). Distributed
+under the Swift Open License v1.0; commercial use above its revenue
+threshold requires the Swift Enterprise License.
+
+Rebuild from scratch with the syv checkout's `drafter/` pipeline
+(`gen_data.py` → `capture.py` → GPTQ heads → `build_draft_vocab.py`) over a
+Swift-weighted prompt corpus; `swift_draft_vocab_ids.json` here is the
+exact id list this model serves.