Update README with Q8 engine requirements and upload status notice
Browse files
README.md
CHANGED
|
@@ -15,15 +15,54 @@ tags:
|
|
| 15 |
|
| 16 |
# Qwen3.8-27B-Splash-HQ (Native 8-bit)
|
| 17 |
|
| 18 |
-
|
| 19 |
-
|
| 20 |
-
|
| 21 |
-
|
| 22 |
-
|
| 23 |
-
|
| 24 |
-
|
| 25 |
-
|
| 26 |
-
|
| 27 |
-
|
| 28 |
-
|
| 29 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
|
| 16 |
# Qwen3.8-27B-Splash-HQ (Native 8-bit)
|
| 17 |
|
| 18 |
+
> [!IMPORTANT]
|
| 19 |
+
> ### ⚠️ Runtime Package Compatibility & Upload Status
|
| 20 |
+
> 1. **Weight Upload in Progress:** The 27 GB model weight files are currently streaming to this repository (~35% complete, ETA ~1.5h). The repository initially only contained tokenizer configurations and metadata.
|
| 21 |
+
> 2. **Engine Requirements:** This model is packed in the **`splash-packed-q8` (schema_version 5)** format developed for high-precision Apple Silicon evaluation.
|
| 22 |
+
> * Official upstream Splash 1.0 (`incoai/splash`) only recognizes 4-bit formats (`splash-packed-q4`, schema 3/4) and will reject Q8 packages with `repository is not a supported Splash runtime package`.
|
| 23 |
+
> * To run this model, use the Q8-enabled Metal runtime fork available at:
|
| 24 |
+
> 👉 **[https://github.com/npanj/qwen3.8-27b-apple-silicon-eval](https://github.com/npanj/qwen3.8-27b-apple-silicon-eval)**
|
| 25 |
+
|
| 26 |
+
---
|
| 27 |
+
|
| 28 |
+
## Overview
|
| 29 |
+
|
| 30 |
+
This repository contains the uncompressed native 8-bit weights of **Qwen3.8-27B** with built-in Multi-Token Prediction (MTP) heads, optimized for high-throughput speculative decoding on Apple Silicon.
|
| 31 |
+
|
| 32 |
+
All 64 layers are packed with true uncompressed 8-bit tiled weights (`MDFL0008`), preserving exact associative reasoning while running speculative verification via Splash's compiled Metal kernel pipeline.
|
| 33 |
+
|
| 34 |
+
## Performance Benchmark vs Other Engines (Apple M5 Pro, 64 GiB)
|
| 35 |
+
|
| 36 |
+
| Model / Engine | Format | Speculative Scheme | Decode Speed (tok/s) | Speedup vs Stock AR |
|
| 37 |
+
| :--- | :--- | :--- | :---: | :---: |
|
| 38 |
+
| **Qwen3.8-27B-Splash-HQ** | **Native 8-bit** | **Splash Metal MTP** | **36.9 t/s** (peak 54.8) | **3.73x** |
|
| 39 |
+
| Splash-Q8 | Transcoded 8-bit | Splash Metal MTP | 36.5 t/s (peak 52.7) | 3.69x |
|
| 40 |
+
| Splash-Q4 | 4-bit | Splash Metal MTP | 60.7 t/s (peak 83.3) | 6.13x |
|
| 41 |
+
| MTPLX-Optimized-Quality | 8-bit | MTPLX DraftCore D3 | 26.5 t/s (peak 28.8) | 2.68x |
|
| 42 |
+
| MLX-Community 8bit | 8-bit | None (Autoregressive) | 9.9 t/s | 1.00x |
|
| 43 |
+
| llama.cpp Q8_0 GGUF | Q8_0 | None (Autoregressive) | 9.9 t/s | 1.00x |
|
| 44 |
+
|
| 45 |
+
---
|
| 46 |
+
|
| 47 |
+
## How to Run
|
| 48 |
+
|
| 49 |
+
### Step 1: Clone the Q8 Splash Engine
|
| 50 |
+
```bash
|
| 51 |
+
git clone https://github.com/npanj/qwen3.8-27b-apple-silicon-eval.git
|
| 52 |
+
cd qwen3.8-27b-apple-silicon-eval
|
| 53 |
+
```
|
| 54 |
+
|
| 55 |
+
### Step 2: Serve the Model
|
| 56 |
+
```bash
|
| 57 |
+
python server/server.py \
|
| 58 |
+
install/models/incoai/Qwen3.8-27B-Splash-HQ/target \
|
| 59 |
+
install/models/incoai/Qwen3.8-27B-Splash-HQ/draft \
|
| 60 |
+
--tokenizer install/models/incoai/Qwen3.8-27B-Splash-HQ/tokenizer \
|
| 61 |
+
--model incoai/Qwen3.8-27B-Splash-HQ \
|
| 62 |
+
--port 8000
|
| 63 |
+
```
|
| 64 |
+
|
| 65 |
+
### Step 3: Connect Coding Agents (Oh My Pi / Claude Code)
|
| 66 |
+
```bash
|
| 67 |
+
omp --model splash/incoai/Qwen3.8-27B-Splash-HQ
|
| 68 |
+
```
|