Add model card

README.md

@@ -0,0 +1,187 @@
+---
+base_model: ukisai/Swift-Qwen3.8-27b
+model_name: Swift-Qwen3.8-27b-oQ4-mtp
+library_name: mlx
+pipeline_tag: image-text-to-text
+license: other
+license_name: swift-open-license-1.0
+license_link: "https://huggingface.co/ukisai/Swift-Qwen3.8-27b#license-and-access"
+tags:
+  - mlx
+  - omlx
+  - quantization
+  - mixed-precision
+  - apple-silicon
+  - mtp
+  - speculative-decoding
+  - qwen
+  - vision
+  - base_model:quantized:ukisai/Swift-Qwen3.8-27b
+  - base_model_size:10B to 100B
+---
+
+# Swift-Qwen3.8-27b-oQ4-mtp — mixed 4/5-bit MLX quant of Swift-Qwen3.8-27b (MTP head kept)
+
+Unofficial Apple Silicon quantization of **[ukisai/Swift-Qwen3.8-27b](https://huggingface.co/ukisai/Swift-Qwen3.8-27b)**, produced with **oMLX 0.6.4** (MLX `affine` quantizer, group size 64) on an Apple M5 Max / 128 GB.
+
+Two things are preserved on purpose:
+
+- **Swift's reasoning efficiency.** Smallest, fastest to load. Best default for coding/agentic work where you want the shortest reasoning and maximum headroom for KV cache.
+- **The MTP head.** The `-mtp` suffix means the multi-token-prediction head from the base checkpoint ships intact (29 tensors, `mtp_num_hidden_layers: 1`), so oMLX can run self-speculative decoding instead of wasting the weights.
+
+I am not affiliated with UkisAI. All upstream weights, benchmarks and license terms belong to UkisAI, and **the upstream license governs this repository too** (see [License](#license)).
+
+> Format note: these are **MLX safetensors**, not GGUF. They will not load in llama.cpp / Ollama / LM Studio. Use oMLX or MLX runtimes.
+
+## Pick a variant
+
+| | oQ4-mtp | oQ6-mtp |
+|---|---|---|
+| Weights on disk | 15.81 GiB (16.97 GB), 4 shards | 22.09 GiB (23.72 GB), 5 shards |
+| Weight precision | mixed 4/5-bit, group 64 | mixed 6/8-bit, group 64 |
+| Effective bits / quantized weight | 4.68 | 6.66 |
+| Effective bits / parameter (whole repo) | 4.89 | 6.83 |
+| Minimum Apple Silicon RAM | 24 GB (context ≲ 32k) | 32 GB (short context) |
+| Comfortable | 32 GB+ | 48 GB+ |
+| Best for | memory-bound, long agentic sessions | max fidelity, math, vision |
+
+Smallest, fastest to load. Best default for coding/agentic work where you want the shortest reasoning and maximum headroom for KV cache.
+
+## What is inside (read straight from the shipped `config.json`)
+
+| Field | Value |
+|---|---|
+| Architecture | `Qwen3_5ForConditionalGeneration` (`model_type: qwen3_5`) |
+| Parameters | 27.78 B total, 27.27 B quantized (98.1%) |
+| Text layers / hidden | 64 layers, `hidden_size` 5120, `intermediate_size` 17408 |
+| Attention | hybrid: 1 full-attention layer every 4 (`full_attention_interval: 4`), 24 heads / 4 KV, `head_dim` 256, `attn_output_gate: true`; the rest are gated linear-attention (`linear_attn`) |
+| Context | `max_position_embeddings: 262144` |
+| Vocab | 248,320 (tokenizer and `chat_template.jinja` copied from upstream, unchanged) |
+| MTP | `mtp_num_hidden_layers: 1`, `mtp_use_dedicated_embeddings: false`, weights included |
+| Vision | Qwen vision tower kept in **BF16** (~0.92 GB), depth 27, patch 16, spatial merge 2 |
+| Metadata | `{"format": "mlx"}` in every safetensors header |
+
+## Quantization recipe
+
+Precision is mixed per module and recorded verbatim in `config.json` → `quantization_config`, so any MLX loader reproduces the layout without guessing:
+
+- default: **mixed**, `group_size: 64`, `mode: affine`
+- 339 modules @ 4-bit + 166 modules bumped to 5-bit
+- bumped modules: early layers (`linear_attn.in_proj_a/b/z`, `linear_attn.out_proj`, `mlp.down_proj`, some `self_attn.k_proj/o_proj`)
+- **never quantized:** vision tower (BF16), all `scales`/`biases` (1.68 GB BF16), norms, `A_log`, `dt_bias`, convolutions (≈5 MB), MTP non-linear weights (128 MB)
+
+## Requirements
+
+- Apple Silicon, macOS 15+ (built and smoke-tested on M5 Max, 128 GB)
+- oMLX **≥ 0.6.4**, or a recent `mlx` / `mlx-lm` / `mlx-vlm` build with `qwen3_5` support
+
+## Usage
+
+### oMLX (the runtime these were made for)
+
+```bash
+# 1. drop the folder into the oMLX model dir
+git clone https://huggingface.co/suzu89/Swift-Qwen3.8-27b-oQ4-mtp ~/.omlx/models/Swift-Qwen3.8-27b-oQ4-mtp
+
+# 2. start the multi-model server (model id = folder name)
+omlx serve --model-dir ~/.omlx/models --port 8000
+
+# 3. talk to it
+curl -s http://127.0.0.1:8000/v1/chat/completions \
+  -H 'Content-Type: application/json' \
+  -d '{"model": "Swift-Qwen3.8-27b-oQ4-mtp",
+       "messages": [{"role": "user", "content": "Explain speculative decoding in two sentences."}]}'
+```
+
+In oMLX model settings, enable the speculative head and the matching reasoning parser:
+
+```json
+{
+  "mtp_enabled": true,
+  "reasoning_parser": "qwen_3_5",
+  "max_context_window": 262144,
+  "model_type_override": "vlm"
+}
+```
+
+### MLX directly
+
+```bash
+pip install -U mlx-lm mlx-vlm
+python -m mlx_lm.server --model suzu89/Swift-Qwen3.8-27b-oQ4-mtp --port 8000
+```
+
+Only oMLX 0.6.4 is verified by me; if you get `mlx_lm` running this architecture, please open an issue and I will document it.
+
+### Not supported
+
+`llama.cpp`, GGUF, vLLM and SGLang paths in the upstream card do not apply here — this repo has no GGUF and no PyTorch weights. For BF16/server deployments use `ukisai/Swift-Qwen3.8-27b`.
+
+## Recommended sampling
+
+Shipped `generation_config.json` (unchanged from upstream) is the tuning target for thinking mode:
+
+```
+temperature 1.0 · top_p 0.95 · top_k 20 · min_p 0.0 · repetition_penalty 1.0
+eos_token_id [248046, 248044]
+```
+
+The upstream chat template supports tool calling, image/video inputs and an `enable_thinking` switch, so you can trade reasoning length per request; upstream reports Swift's token savings hold at `xhigh`, `medium` and `low` reasoning effort.
+
+## Benchmarks
+
+I publish no numbers I have not measured myself. This table is the honest state of the repository:
+
+| Benchmark | oQ4-mtp | oQ6-mtp | BF16 upstream (reference) |
+|---|---|---|---|
+| GPQA-Diamond | not measured | not measured | 88.28% |
+| AIME 2026 | not measured | not measured | 94.00% |
+| LiveCodeBench v6 | not measured | not measured | 81.55% |
+| IFBench | not measured | not measured | 71.80% |
+
+Upstream Swift vs Qwen3.8-27B results (GPQA-Diamond −0.1 pt for ~41% fewer mean thinking tokens, ~1.95× faster) are reported by UkisAI in the [base model card](https://huggingface.co/ukisai/Swift-Qwen3.8-27b) and are **not** measurements of these quantized weights. What you should realistically expect from a quantization of a "think less" fine-tune: token savings largely survive (they come from behaviour, not precision), while the hardest math splits and long-horizon tool chains degrade slightly — most visibly at 4-bit.
+
+Measured throughput/acceptance-length data and issue reports (especially "quant X broke task Y") are welcome and will be merged into this table.
+
+## Known caveats
+
+- Quantization is lossy: expect small regressions versus BF16, largest on competition math and very long agentic traces. Try `oQ6-mtp` before filing a bug.
+- `oQ4-mtp` can amplify repetition on degenerate loops; keep `repetition_penalty` at 1.0 first and only then nudge it.
+- Vision works through the BF16 tower, but I have not benchmarked VQA accuracy post-quantization.
+- 262k context is the architecture's limit, not a promise: keep KV cache within your memory budget or the system swaps.
+- MTP decoding only helps when the speculative draft is enabled in the runtime; without it you pay for the head and get nothing.
+
+## License
+
+**This repository is distributed under the [Swift Open License v1.0](https://huggingface.co/ukisai/Swift-Qwen3.8-27b#license-and-access).** A quantization is a derivative work: it inherits the upstream terms in full and cannot be released under a more permissive license.
+
+- Free personal, research, educational, evaluation and commercial use for individuals and organizations with annual recurring revenue (including affiliates) **up to US$1,000,000**.
+- Above that threshold, commercial use requires a separate **Swift Enterprise License** from UkisAI.
+- The base Qwen3.8 checkpoint and the ThinkingCap-Qwen3.6-27B transfer component (BottleCap AI) contribute their own terms, which apply to you as well — read the `LICENSE` files in [`ukisai/Swift-Qwen3.8-27b`](https://huggingface.co/ukisai/Swift-Qwen3.8-27b) and the BottleCap repository before commercial deployment.
+- Keep this attribution, the upstream citation and the `base_model` metadata intact when you redistribute.
+
+## Citation
+
+```bibtex
+@misc{swift-qwen3.8-27b-mlx-quants,
+  title   = {Swift-Qwen3.8-27b-oQ4-mtp}: oMLX/MLX quantization of Swift-Qwen3.8-27B with MTP head retained,
+  author  = {suzu89},
+  year    = {2026},
+  howpublished = {\url{https://huggingface.co/suzu89/Swift-Qwen3.8-27b-oQ4-mtp}},
+  note    = {Unofficial quantization of ukisai/Swift-Qwen3.8-27b}
+}
+
+@misc{swift-qwen3.8-27b,
+  title  = {Swift-Qwen3.8-27B},
+  author = {UkisAI},
+  year   = {2026},
+  url    = {https://huggingface.co/ukisai/Swift-Qwen3.8-27b}
+}
+```
+
+## Acknowledgements
+
+- **UkisAI** for Swift-Qwen3.8-27B and the public evaluation harness.
+- **Qwen team** for the Qwen3.8-27B base model.
+- **BottleCap AI** for the ThinkingCap-Qwen3.6-27B transfer component used upstream.
+- **oMLX** for the Apple Silicon server and quantizer that made these builds possible.