Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -75,7 +75,7 @@ cmake --build build --target llama-cli llama-server
|
|
| 75 |
|
| 76 |
```bash
|
| 77 |
llama-server -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
|
| 78 |
-
--spec-type mtp --spec-draft-n-max
|
| 79 |
--cache-type-k q4_0 --cache-type-v q4_0 \
|
| 80 |
-np 1 -c 262144 --temp 0.7 --top-k 20 -ngl 99 --port 8081
|
| 81 |
```
|
|
@@ -84,7 +84,7 @@ That's it. Three optimizations in one command:
|
|
| 84 |
|
| 85 |
| Flag | What it does | Impact |
|
| 86 |
|---|---|---|
|
| 87 |
-
| `--spec-type mtp --spec-draft-n-max
|
| 88 |
| `--cache-type-k q4_0 --cache-type-v q4_0` | 4-bit KV cache (instead of 16-bit) | **Quarter the KV memory** |
|
| 89 |
| `-c 262144` | 262K context window | Full native context on **48 GB Mac** with q4_0 KV |
|
| 90 |
|
|
@@ -225,7 +225,7 @@ curl http://localhost:8081/v1/messages \
|
|
| 225 |
```bash
|
| 226 |
# Text generation
|
| 227 |
llama-cli -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
|
| 228 |
-
--spec-type mtp --spec-draft-n-max
|
| 229 |
--cache-type-k q8_0 --cache-type-v q8_0 \
|
| 230 |
-np 1 -c 4096 -n 2048 --temp 0.7 -ngl 99 \
|
| 231 |
-p "Your prompt here"
|
|
@@ -270,12 +270,12 @@ Effect on hardware requirements (Q5_K_M, 80K context):
|
|
| 270 |
The model predicts 5 extra tokens per step using its own MTP heads, then verifies them in one pass. No extra model needed.
|
| 271 |
|
| 272 |
```bash
|
| 273 |
-
--spec-type mtp --spec-draft-n-max
|
| 274 |
```
|
| 275 |
|
| 276 |
MTP currently requires `-np 1` (single-sequence mode). Without it, you'll get: `MTP currently supports only n_parallel=1`.
|
| 277 |
|
| 278 |
-
Tune `--spec-draft-n-max`: **
|
| 279 |
|
| 280 |
### Draft model (~2.3x faster)
|
| 281 |
|
|
|
|
| 75 |
|
| 76 |
```bash
|
| 77 |
llama-server -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
|
| 78 |
+
--spec-type mtp --spec-draft-n-max 3 \
|
| 79 |
--cache-type-k q4_0 --cache-type-v q4_0 \
|
| 80 |
-np 1 -c 262144 --temp 0.7 --top-k 20 -ngl 99 --port 8081
|
| 81 |
```
|
|
|
|
| 84 |
|
| 85 |
| Flag | What it does | Impact |
|
| 86 |
|---|---|---|
|
| 87 |
+
| `--spec-type mtp --spec-draft-n-max 3` | Multi-Token Prediction (built into the model) | **2.5x faster** generation |
|
| 88 |
| `--cache-type-k q4_0 --cache-type-v q4_0` | 4-bit KV cache (instead of 16-bit) | **Quarter the KV memory** |
|
| 89 |
| `-c 262144` | 262K context window | Full native context on **48 GB Mac** with q4_0 KV |
|
| 90 |
|
|
|
|
| 225 |
```bash
|
| 226 |
# Text generation
|
| 227 |
llama-cli -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
|
| 228 |
+
--spec-type mtp --spec-draft-n-max 3 \
|
| 229 |
--cache-type-k q8_0 --cache-type-v q8_0 \
|
| 230 |
-np 1 -c 4096 -n 2048 --temp 0.7 -ngl 99 \
|
| 231 |
-p "Your prompt here"
|
|
|
|
| 270 |
The model predicts 5 extra tokens per step using its own MTP heads, then verifies them in one pass. No extra model needed.
|
| 271 |
|
| 272 |
```bash
|
| 273 |
+
--spec-type mtp --spec-draft-n-max 3 -np 1
|
| 274 |
```
|
| 275 |
|
| 276 |
MTP currently requires `-np 1` (single-sequence mode). Without it, you'll get: `MTP currently supports only n_parallel=1`.
|
| 277 |
|
| 278 |
+
Tune `--spec-draft-n-max`: **3** is optimal for general use (83% acceptance rate). Values of 1–2 are more conservative; 4–5 waste compute on rejected tokens.
|
| 279 |
|
| 280 |
### Draft model (~2.3x faster)
|
| 281 |
|