nitinpanj commited on
Commit
6583723
·
verified ·
1 Parent(s): 4cfd7fb

Update README with Q8 engine requirements and upload status notice

Browse files
Files changed (1) hide show
  1. README.md +51 -12
README.md CHANGED
@@ -15,15 +15,54 @@ tags:
15
 
16
  # Qwen3.8-27B-Splash-HQ (Native 8-bit)
17
 
18
- This repository contains the uncompressed native 8-bit weights of **Qwen3.8-27B** with built-in Multi-Token Prediction (MTP) heads, optimized for the **Splash** inference engine on Apple Silicon.
19
-
20
- ## Performance vs MTPLX & MLX on Apple Silicon
21
-
22
- | Model | Format | Decode Speed (tok/s) | Speedup vs Stock AR |
23
- | :--- | :--- | :---: | :---: |
24
- | **Qwen3.8-27B-Splash-HQ** | **Native 8-bit + MTP** | **36.9 t/s** (peak 54.8) | **3.73x** |
25
- | Splash-Q8 | Compressed 8-bit | 36.5 t/s (peak 52.7) | 3.69x |
26
- | Splash-Q4 | 4-bit + MTP | 60.7 t/s (peak 83.3) | 6.13x |
27
- | MTPLX-Optimized-Quality | 8-bit + MTP D3 | 26.5 t/s (peak 28.8) | 2.68x |
28
- | MLX-Community 8bit | 8-bit Stock AR | 9.9 t/s | 1.00x |
29
- | llama.cpp Q8_0 GGUF | Q8_0 Stock AR | 9.9 t/s | 1.00x |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
15
 
16
  # Qwen3.8-27B-Splash-HQ (Native 8-bit)
17
 
18
+ > [!IMPORTANT]
19
+ > ### ⚠️ Runtime Package Compatibility & Upload Status
20
+ > 1. **Weight Upload in Progress:** The 27 GB model weight files are currently streaming to this repository (~35% complete, ETA ~1.5h). The repository initially only contained tokenizer configurations and metadata.
21
+ > 2. **Engine Requirements:** This model is packed in the **`splash-packed-q8` (schema_version 5)** format developed for high-precision Apple Silicon evaluation.
22
+ > * Official upstream Splash 1.0 (`incoai/splash`) only recognizes 4-bit formats (`splash-packed-q4`, schema 3/4) and will reject Q8 packages with `repository is not a supported Splash runtime package`.
23
+ > * To run this model, use the Q8-enabled Metal runtime fork available at:
24
+ > 👉 **[https://github.com/npanj/qwen3.8-27b-apple-silicon-eval](https://github.com/npanj/qwen3.8-27b-apple-silicon-eval)**
25
+
26
+ ---
27
+
28
+ ## Overview
29
+
30
+ This repository contains the uncompressed native 8-bit weights of **Qwen3.8-27B** with built-in Multi-Token Prediction (MTP) heads, optimized for high-throughput speculative decoding on Apple Silicon.
31
+
32
+ All 64 layers are packed with true uncompressed 8-bit tiled weights (`MDFL0008`), preserving exact associative reasoning while running speculative verification via Splash's compiled Metal kernel pipeline.
33
+
34
+ ## Performance Benchmark vs Other Engines (Apple M5 Pro, 64 GiB)
35
+
36
+ | Model / Engine | Format | Speculative Scheme | Decode Speed (tok/s) | Speedup vs Stock AR |
37
+ | :--- | :--- | :--- | :---: | :---: |
38
+ | **Qwen3.8-27B-Splash-HQ** | **Native 8-bit** | **Splash Metal MTP** | **36.9 t/s** (peak 54.8) | **3.73x** |
39
+ | Splash-Q8 | Transcoded 8-bit | Splash Metal MTP | 36.5 t/s (peak 52.7) | 3.69x |
40
+ | Splash-Q4 | 4-bit | Splash Metal MTP | 60.7 t/s (peak 83.3) | 6.13x |
41
+ | MTPLX-Optimized-Quality | 8-bit | MTPLX DraftCore D3 | 26.5 t/s (peak 28.8) | 2.68x |
42
+ | MLX-Community 8bit | 8-bit | None (Autoregressive) | 9.9 t/s | 1.00x |
43
+ | llama.cpp Q8_0 GGUF | Q8_0 | None (Autoregressive) | 9.9 t/s | 1.00x |
44
+
45
+ ---
46
+
47
+ ## How to Run
48
+
49
+ ### Step 1: Clone the Q8 Splash Engine
50
+ ```bash
51
+ git clone https://github.com/npanj/qwen3.8-27b-apple-silicon-eval.git
52
+ cd qwen3.8-27b-apple-silicon-eval
53
+ ```
54
+
55
+ ### Step 2: Serve the Model
56
+ ```bash
57
+ python server/server.py \
58
+ install/models/incoai/Qwen3.8-27B-Splash-HQ/target \
59
+ install/models/incoai/Qwen3.8-27B-Splash-HQ/draft \
60
+ --tokenizer install/models/incoai/Qwen3.8-27B-Splash-HQ/tokenizer \
61
+ --model incoai/Qwen3.8-27B-Splash-HQ \
62
+ --port 8000
63
+ ```
64
+
65
+ ### Step 3: Connect Coding Agents (Oh My Pi / Claude Code)
66
+ ```bash
67
+ omp --model splash/incoai/Qwen3.8-27B-Splash-HQ
68
+ ```