---
base_model: ukisai/Swift-Qwen3.8-27b
model_name: Swift-Qwen3.8-27b-oQ4-mtp
library_name: mlx
pipeline_tag: image-text-to-text
license: other
license_name: swift-open-license-1.0
license_link: "https://huggingface.co/ukisai/Swift-Qwen3.8-27b#license-and-access"
tags:
  - mlx
  - omlx
  - quantization
  - mixed-precision
  - apple-silicon
  - mtp
  - speculative-decoding
  - qwen
  - vision
  - base_model:quantized:ukisai/Swift-Qwen3.8-27b
  - base_model_size:10B to 100B
---

# Swift-Qwen3.8-27b-oQ4-mtp — mixed 4/5-bit MLX quant of Swift-Qwen3.8-27b (MTP head kept)

Unofficial Apple Silicon quantization of **[ukisai/Swift-Qwen3.8-27b](https://huggingface.co/ukisai/Swift-Qwen3.8-27b)**, produced with **oMLX 0.6.4** (MLX `affine` quantizer, group size 64) on an Apple M5 Max / 128 GB.

Two things are preserved on purpose:

- **Swift weights.** This variant quantizes the upstream Swift checkpoint; task-level quality after quantization has not been measured here.
- **The MTP head.** The `-mtp` suffix means the multi-token-prediction head from the base checkpoint ships intact (29 tensors, `mtp_num_hidden_layers: 1`), so oMLX can run self-speculative decoding instead of wasting the weights.

I am not affiliated with UkisAI. All upstream weights, benchmarks and license terms belong to UkisAI, and **the upstream license governs this repository too** (see [License](#license)).

> Format note: these are **MLX safetensors**, not GGUF. They will not load in llama.cpp / Ollama / LM Studio. Use oMLX or MLX runtimes.

## Pick a variant

| | [oQ4-mtp](https://huggingface.co/TokenAI-zer/Swift-Qwen3.8-27b-oQ4-mtp) | [oQ6-mtp](https://huggingface.co/TokenAI-zer/Swift-Qwen3.8-27b-oQ6-mtp) | [oQ8-mtp](https://huggingface.co/TokenAI-zer/Swift-Qwen3.8-27b-oQ8-mtp) |
|---|---|---|---|
| Weights on disk | 15.81 GiB (16.97 GB), 4 shards | 22.09 GiB (23.72 GB), 5 shards | 27.94 GiB (30.00 GB), 6 shards |
| Weight precision | mixed 4/5-bit | mixed 6/8-bit | uniform 8-bit |
| Quantization | affine, group size 64 | affine, group size 64 | affine, group size 64 |
| Choose when | prioritize memory headroom | balance size and weight precision | prioritize weight precision and have more memory |

All three retain the MTP head. Task-level accuracy and speed after quantization have not been measured here. Runtime memory also depends on context length and KV cache.

## What is inside (read straight from the shipped `config.json`)

| Field | Value |
|---|---|
| Architecture | `Qwen3_5ForConditionalGeneration` (`model_type: qwen3_5`) |
| Parameters | 27.78 B total, 27.27 B quantized (98.1%) |
| Text layers / hidden | 64 layers, `hidden_size` 5120, `intermediate_size` 17408 |
| Attention | hybrid: 1 full-attention layer every 4 (`full_attention_interval: 4`), 24 heads / 4 KV, `head_dim` 256, `attn_output_gate: true`; the rest are gated linear-attention (`linear_attn`) |
| Context | `max_position_embeddings: 262144` |
| Vocab | 248,320 (tokenizer and `chat_template.jinja` copied from upstream, unchanged) |
| MTP | `mtp_num_hidden_layers: 1`, `mtp_use_dedicated_embeddings: false`, weights included |
| Vision | Qwen vision tower kept in **BF16** (~0.92 GB), depth 27, patch 16, spatial merge 2 |
| Metadata | `{"format": "mlx"}` in every safetensors header |

## Quantization recipe

Precision is mixed per module and recorded verbatim in `config.json` → `quantization_config`, so any MLX loader reproduces the layout without guessing:

- default: **mixed**, `group_size: 64`, `mode: affine`
- 339 modules @ 4-bit + 166 modules bumped to 5-bit
- bumped modules: early layers (`linear_attn.in_proj_a/b/z`, `linear_attn.out_proj`, `mlp.down_proj`, some `self_attn.k_proj/o_proj`)
- **never quantized:** vision tower (BF16), all `scales`/`biases` (1.68 GB BF16), norms, `A_log`, `dt_bias`, convolutions (≈5 MB), MTP non-linear weights (128 MB)

## Requirements

- Apple Silicon, macOS 15+ (built and smoke-tested on M5 Max, 128 GB)
- oMLX **≥ 0.6.4**, or a recent `mlx` / `mlx-lm` / `mlx-vlm` build with `qwen3_5` support

## Usage

### oMLX (the runtime these were made for)

```bash
# 1. drop the folder into the oMLX model dir
git clone https://huggingface.co/TokenAI-zer/Swift-Qwen3.8-27b-oQ4-mtp ~/.omlx/models/Swift-Qwen3.8-27b-oQ4-mtp

# 2. start the multi-model server (model id = folder name)
omlx serve --model-dir ~/.omlx/models --port 8000

# 3. talk to it
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "Swift-Qwen3.8-27b-oQ4-mtp",
       "messages": [{"role": "user", "content": "Explain speculative decoding in two sentences."}]}'
```

In oMLX model settings, enable the speculative head and the matching reasoning parser:

```json
{
  "mtp_enabled": true,
  "reasoning_parser": "qwen_3_5",
  "max_context_window": 262144,
  "model_type_override": "vlm"
}
```

### MLX directly

```bash
pip install -U mlx-lm mlx-vlm
python -m mlx_lm.server --model TokenAI-zer/Swift-Qwen3.8-27b-oQ4-mtp --port 8000
```

Only oMLX 0.6.4 is verified by me; if you get `mlx_lm` running this architecture, please open an issue and I will document it.

### Not supported

`llama.cpp`, GGUF, vLLM and SGLang paths in the upstream card do not apply here — this repo has no GGUF and no PyTorch weights. For BF16/server deployments use `ukisai/Swift-Qwen3.8-27b`.

## Recommended sampling

Shipped `generation_config.json` (unchanged from upstream) is the tuning target for thinking mode:

```
temperature 1.0 · top_p 0.95 · top_k 20 · min_p 0.0 · repetition_penalty 1.0
eos_token_id [248046, 248044]
```

The upstream chat template supports tool calling, image/video inputs and an `enable_thinking` switch, so you can trade reasoning length per request; upstream reports Swift's token savings hold at `xhigh`, `medium` and `low` reasoning effort.

## Benchmarks

I publish no numbers I have not measured myself. This table is the honest state of the repository:

| Benchmark | oQ4-mtp | oQ6-mtp | oQ8-mtp | BF16 upstream (reference) |
|---|---|---|---|---|
| GPQA-Diamond | not measured | not measured | not measured | 88.28% |
| AIME 2026 | not measured | not measured | not measured | 94.00% |
| LiveCodeBench v6 | not measured | not measured | not measured | 81.55% |
| IFBench | not measured | not measured | not measured | 71.80% |

Upstream Swift vs Qwen3.8-27B results (GPQA-Diamond −0.1 pt for ~41% fewer mean thinking tokens, ~1.95× faster) are reported by UkisAI in the [base model card](https://huggingface.co/ukisai/Swift-Qwen3.8-27b) and are **not** measurements of these quantized weights. What you should realistically expect from a quantization of a "think less" fine-tune: token savings largely survive (they come from behaviour, not precision), while the hardest math splits and long-horizon tool chains degrade slightly — most visibly at 4-bit.

Measured throughput/acceptance-length data and issue reports (especially "quant X broke task Y") are welcome and will be merged into this table.

## Known caveats

- Quantization is lossy. If you see a regression, compare with a higher precision variant and the BF16 source before filing a bug.
- `oQ4-mtp` can amplify repetition on degenerate loops; keep `repetition_penalty` at 1.0 first and only then nudge it.
- Vision works through the BF16 tower, but I have not benchmarked VQA accuracy post-quantization.
- 262k context is the architecture's limit, not a promise: keep KV cache within your memory budget or the system swaps.
- MTP decoding only helps when the speculative draft is enabled in the runtime; without it you pay for the head and get nothing.

## License

**This repository is distributed under the [Swift Open License v1.0](https://huggingface.co/ukisai/Swift-Qwen3.8-27b#license-and-access).** A quantization is a derivative work: it inherits the upstream terms in full and cannot be released under a more permissive license.

- Free personal, research, educational, evaluation and commercial use for individuals and organizations with annual recurring revenue (including affiliates) **up to US$1,000,000**.
- Above that threshold, commercial use requires a separate **Swift Enterprise License** from UkisAI.
- The base Qwen3.8 checkpoint and the ThinkingCap-Qwen3.6-27B transfer component (BottleCap AI) contribute their own terms, which apply to you as well — read the `LICENSE` files in [`ukisai/Swift-Qwen3.8-27b`](https://huggingface.co/ukisai/Swift-Qwen3.8-27b) and the BottleCap repository before commercial deployment.
- Keep this attribution, the upstream citation and the `base_model` metadata intact when you redistribute.

## Citation

```bibtex
@misc{swift-qwen3.8-27b-mlx-quants,
  title   = {Swift-Qwen3.8-27b-oQ4-mtp}: oMLX/MLX quantization of Swift-Qwen3.8-27B with MTP head retained,
  author  = {TokenAI-zer},
  year    = {2026},
  howpublished = {\url{https://huggingface.co/TokenAI-zer/Swift-Qwen3.8-27b-oQ4-mtp}},
  note    = {Unofficial quantization of ukisai/Swift-Qwen3.8-27b}
}

@misc{swift-qwen3.8-27b,
  title  = {Swift-Qwen3.8-27B},
  author = {UkisAI},
  year   = {2026},
  url    = {https://huggingface.co/ukisai/Swift-Qwen3.8-27b}
}
```

## Acknowledgements

- **UkisAI** for Swift-Qwen3.8-27B and the public evaluation harness.
- **Qwen team** for the Qwen3.8-27B base model.
- **BottleCap AI** for the ThinkingCap-Qwen3.6-27B transfer component used upstream.
- **oMLX** for the Apple Silicon server and quantizer that made these builds possible.