| --- |
| license: apache-2.0 |
| language: |
| - en |
| - zh |
| pipeline_tag: text-generation |
| tags: |
| - apple-silicon |
| - metal |
| - splash |
| - mtp |
| - speculative-decoding |
| - 8bit |
| --- |
| |
| # Qwen3.8-27B-Splash-HQ (Native 8-bit) |
|
|
| > [!IMPORTANT] |
| > ### ⚠️ Runtime Package Compatibility Notice |
| > This model is packed in the **`splash-packed-q8` (schema_version 5)** format developed for high-precision Apple Silicon evaluation. |
| > * Official upstream Splash 1.0 (`incoai/splash`) only recognizes 4-bit formats (`splash-packed-q4`, schema 3/4) and will reject Q8 packages with `repository is not a supported Splash runtime package`. |
| > * To run this model, use the Q8-enabled Metal runtime fork available at: |
| > 👉 **[https://github.com/npanj/splash/tree/q8](https://github.com/npanj/splash/tree/q8)** |
| > |
| > All 81 model files (target layers, embedding, head, draft model, vision model, tokenizer, and manifest) are 100% complete and verified. |
| |
| --- |
| |
| ## Overview |
| |
| This repository contains the uncompressed native 8-bit weights of **Qwen3.8-27B** with built-in Multi-Token Prediction (MTP) heads, optimized for high-throughput speculative decoding on Apple Silicon. |
| |
| All 64 layers are packed with true uncompressed 8-bit tiled weights (`MDFL0008`), preserving exact associative reasoning while running speculative verification via Splash's compiled Metal kernel pipeline. |
| |
| ## Performance Benchmark vs Other Engines (Apple M-Series, 64 GiB) |
| |
| | Model / Engine | Format | Speculative Scheme | Decode Speed (tok/s) | Speedup vs Stock AR | |
| | :--- | :--- | :--- | :---: | :---: | |
| | **Qwen3.8-27B-Splash-HQ** | **Native 8-bit** | **Splash Metal MTP** | **36.9 t/s** (peak 54.8) | **3.73x** | |
| | Splash-Q8 | Transcoded 8-bit | Splash Metal MTP | 36.5 t/s (peak 52.7) | 3.69x | |
| | Splash-Q4 | 4-bit | Splash Metal MTP | 60.7 t/s (peak 83.3) | 6.13x | |
| | MTPLX-Optimized-Quality | 8-bit | MTPLX DraftCore D3 | 26.5 t/s (peak 28.8) | 2.68x | |
| | MLX-Community 8bit | 8-bit | None (Autoregressive) | 9.9 t/s | 1.00x | |
| | llama.cpp Q8_0 GGUF | Q8_0 | None (Autoregressive) | 9.9 t/s | 1.00x | |
| |
| --- |
| |
| ## Step-by-Step Setup Recipe |
| |
| ### 1. Clone the Q8 Splash Engine Fork |
| ```bash |
| git clone https://github.com/npanj/splash.git -b q8 |
| cd splash |
| ``` |
| |
| ### 2. Build the Metal Kernels |
| ```bash |
| make -j4 |
| ``` |
| |
| ### 3. Serve the Model |
| ```bash |
| ./splash serve --model nitinpanj/Qwen3.8-27B-Splash-HQ --port 8000 |
| ``` |
| *(On first launch, this automatically verifies and downloads the model into `install/models/nitinpanj/Qwen3.8-27B-Splash-HQ` and starts the OpenAI-compatible API on `http://127.0.0.1:8000`).* |
| |
| ### 4. Connect Coding Agents |
| ```bash |
| # Connect with Oh My Pi (OMP) |
| omp --model splash/nitinpanj/Qwen3.8-27B-Splash-HQ |
| |
| # Or connect with any OpenAI-compatible client at http://127.0.0.1:8000/v1 |
| ``` |
| |
| For the complete benchmark report, evaluation scripts, and raw JSON logs: |
| 👉 **[https://github.com/npanj/splash-plus](https://github.com/npanj/splash-plus)** |
| |
