---
license: other
license_name: swift-open-license-1.0
library_name: transformers
pipeline_tag: image-text-to-text
base_model: ukisai/Swift-Qwen3.8-27b
tags:
  - qwen
  - qwen3.5
  - qwen3.8
  - finetune
  - awq
  - int4
  - w4a16
  - compressed-tensors
  - lmdeploy
---

# Swift-Qwen3.8-27B-W4A16-AWQ

W4A16 (4-bit weights, 16-bit activations) AWQ compressed-tensors quantization
of [`ukisai/Swift-Qwen3.8-27b`](https://huggingface.co/ukisai/Swift-Qwen3.8-27b).

## Quantization method

- **Scheme:** `W4A16_ASYM` — 4-bit asymmetric per-group quantization (group size 128) of all `Linear` weights, stored in the compressed-tensors pack-quantized format (`weight_packed` / `weight_scale` / `weight_zero_point` / `weight_shape`), which LMDeploy `turbomind` auto-detects and loads natively (including the MTP head and vision tower, which stay BF16).
- **Tooling:** [llmcompressor](https://github.com/vllm-project/llmcompressor) one-shot offline quantization with CPU offloading (`compressed_tensors.offload.load_offloaded_model`), so the full-precision source fits on a 2×16 GB VRAM setup.
- **AWQ activation smoothing:** `AWQModifier` with the layer-scoped hybrid-attention mappings from `build_hybrid_attention_mappings` — full-attention `input_layernorm` → `self_attn.q/k/v`, `post_attention_layernorm` → `mlp.gate/up`, and `mlp.up_proj` → `mlp.down_proj`, with `duo_scaling="both"` and CPU offload, followed by W4A16 quantization. This layer-scoped recipe is required for hybrid-attention (Qwen3.5-family) architectures — grouped-regex smoothing or mismatched mappings corrupt decoding.
- Unquantized (kept BF16): embeddings, `lm_head`, norms, `linear_attn.in_proj_a/b`, the vision tower, and the MTP head.
- **Run command:**

```bash
CUDA_VISIBLE_DEVICES=1,2 python3 quantize-awq-hybrid.py \
  --model_path ./Swift-Qwen3.8-27b \
  --quant_path ./Swift-Qwen3.8-27b-W4A16-AWQ \
  --offload_dir ./Swift-Qwen3.8-27b-W4A16-AWQ
```

(Default calibration: UltraChat 200k `train_sft`.)

## Recommended parameters

From the [base model](https://huggingface.co/ukisai/Swift-Qwen3.8-27b): temperature 1.0, top_p 0.95, top_k 20, min_p 0, presence_penalty 0, repetition_penalty 1.0.

## Usage

Tested with LMDeploy `turbomind`:

```bash
from lmdeploy import pipeline, TurbomindEngineConfig

pipe = pipeline(
    "TheUnderscore/Swift-Qwen3.8-27b-W4A16-AWQ",
    backend_config=TurbomindEngineConfig(
        tp=2,
        model_format="compressed-tensors",
        language_model_only=True,
    ),
)
print(pipe("Hello, who are you?").text)
```

## License and access

Swift weights are distributed through gated access under the **Swift Open License v1.0**;
this quantization inherits that license from the base model. Personal, research,
educational, evaluation, and commercial use are free for individuals and organizations
with annual recurring revenue, including affiliates, of up to US$1,000,000. Above that
threshold, commercial use requires a separate **Swift Enterprise License**. Contact
[UkisAI](https://ukisai.com/contact) for terms.

## Files

- `quantize-awq-hybrid.py` — the script used to produce this quantization (CPU-offloaded, DDP/torchrun, produces properly numbered `-of-N` shards). `--offload_dir` selects where per-rank CPU offload temp folders live (defaults to the current working directory).
- `model-nonquant.safetensors` — unquantized tensors (`mtp.*` and `model.visual.*`) preserved BF16 so the full model architecture is loadable.