File size: 2,893 Bytes
ffd5b9b 6583723 0d3902a 6583723 0d3902a 6583723 899b1bd 6583723 899b1bd 6583723 899b1bd 6583723 899b1bd 6583723 0d3902a 6583723 899b1bd 6583723 0d3902a 899b1bd 0d3902a 899b1bd 0d3902a 6583723 0d3902a 50abb0e | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | ---
license: apache-2.0
language:
- en
- zh
pipeline_tag: text-generation
tags:
- apple-silicon
- metal
- splash
- mtp
- speculative-decoding
- 8bit
---
# Qwen3.8-27B-Splash-HQ (Native 8-bit)
> [!IMPORTANT]
> ### ⚠️ Runtime Package Compatibility Notice
> This model is packed in the **`splash-packed-q8` (schema_version 5)** format developed for high-precision Apple Silicon evaluation.
> * Official upstream Splash 1.0 (`incoai/splash`) only recognizes 4-bit formats (`splash-packed-q4`, schema 3/4) and will reject Q8 packages with `repository is not a supported Splash runtime package`.
> * To run this model, use the Q8-enabled Metal runtime fork available at:
> 👉 **[https://github.com/npanj/splash/tree/q8](https://github.com/npanj/splash/tree/q8)**
>
> All 81 model files (target layers, embedding, head, draft model, vision model, tokenizer, and manifest) are 100% complete and verified.
---
## Overview
This repository contains the uncompressed native 8-bit weights of **Qwen3.8-27B** with built-in Multi-Token Prediction (MTP) heads, optimized for high-throughput speculative decoding on Apple Silicon.
All 64 layers are packed with true uncompressed 8-bit tiled weights (`MDFL0008`), preserving exact associative reasoning while running speculative verification via Splash's compiled Metal kernel pipeline.
## Performance Benchmark vs Other Engines (Apple M-Series, 64 GiB)
| Model / Engine | Format | Speculative Scheme | Decode Speed (tok/s) | Speedup vs Stock AR |
| :--- | :--- | :--- | :---: | :---: |
| **Qwen3.8-27B-Splash-HQ** | **Native 8-bit** | **Splash Metal MTP** | **36.9 t/s** (peak 54.8) | **3.73x** |
| Splash-Q8 | Transcoded 8-bit | Splash Metal MTP | 36.5 t/s (peak 52.7) | 3.69x |
| Splash-Q4 | 4-bit | Splash Metal MTP | 60.7 t/s (peak 83.3) | 6.13x |
| MTPLX-Optimized-Quality | 8-bit | MTPLX DraftCore D3 | 26.5 t/s (peak 28.8) | 2.68x |
| MLX-Community 8bit | 8-bit | None (Autoregressive) | 9.9 t/s | 1.00x |
| llama.cpp Q8_0 GGUF | Q8_0 | None (Autoregressive) | 9.9 t/s | 1.00x |
---
## Step-by-Step Setup Recipe
### 1. Clone the Q8 Splash Engine Fork
```bash
git clone https://github.com/npanj/splash.git -b q8
cd splash
```
### 2. Build the Metal Kernels
```bash
make -j4
```
### 3. Serve the Model
```bash
./splash serve --model nitinpanj/Qwen3.8-27B-Splash-HQ --port 8000
```
*(On first launch, this automatically verifies and downloads the model into `install/models/nitinpanj/Qwen3.8-27B-Splash-HQ` and starts the OpenAI-compatible API on `http://127.0.0.1:8000`).*
### 4. Connect Coding Agents
```bash
# Connect with Oh My Pi (OMP)
omp --model splash/nitinpanj/Qwen3.8-27B-Splash-HQ
# Or connect with any OpenAI-compatible client at http://127.0.0.1:8000/v1
```
For the complete benchmark report, evaluation scripts, and raw JSON logs:
👉 **[https://github.com/npanj/splash-plus](https://github.com/npanj/splash-plus)**
|