Upload README.md with huggingface_hub
Browse files
README.md
CHANGED
|
@@ -77,7 +77,7 @@ cmake --build build --target llama-cli llama-server
|
|
| 77 |
llama-server -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
|
| 78 |
--spec-type mtp --spec-draft-n-max 5 \
|
| 79 |
--cache-type-k q4_0 --cache-type-v q4_0 \
|
| 80 |
-
-c 262144 --temp 0.7 --top-k 20 -ngl 99 --port 8081
|
| 81 |
```
|
| 82 |
|
| 83 |
That's it. Three optimizations in one command:
|
|
@@ -227,7 +227,7 @@ curl http://localhost:8081/v1/messages \
|
|
| 227 |
llama-cli -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
|
| 228 |
--spec-type mtp --spec-draft-n-max 5 \
|
| 229 |
--cache-type-k q8_0 --cache-type-v q8_0 \
|
| 230 |
-
-c 4096 -n 2048 --temp 0.7 -ngl 99 \
|
| 231 |
-p "Your prompt here"
|
| 232 |
|
| 233 |
# Vision (MTP does not work with images — omit --spec-type mtp)
|
|
@@ -270,9 +270,11 @@ Effect on hardware requirements (Q5_K_M, 80K context):
|
|
| 270 |
The model predicts 5 extra tokens per step using its own MTP heads, then verifies them in one pass. No extra model needed.
|
| 271 |
|
| 272 |
```bash
|
| 273 |
-
--spec-type mtp --spec-draft-n-max 5
|
| 274 |
```
|
| 275 |
|
|
|
|
|
|
|
| 276 |
Tune `--spec-draft-n-max`: **5** for general use, **8–10** for code, **3** for creative writing.
|
| 277 |
|
| 278 |
### Draft model (~2.3x faster)
|
|
|
|
| 77 |
llama-server -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
|
| 78 |
--spec-type mtp --spec-draft-n-max 5 \
|
| 79 |
--cache-type-k q4_0 --cache-type-v q4_0 \
|
| 80 |
+
-np 1 -c 262144 --temp 0.7 --top-k 20 -ngl 99 --port 8081
|
| 81 |
```
|
| 82 |
|
| 83 |
That's it. Three optimizations in one command:
|
|
|
|
| 227 |
llama-cli -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
|
| 228 |
--spec-type mtp --spec-draft-n-max 5 \
|
| 229 |
--cache-type-k q8_0 --cache-type-v q8_0 \
|
| 230 |
+
-np 1 -c 4096 -n 2048 --temp 0.7 -ngl 99 \
|
| 231 |
-p "Your prompt here"
|
| 232 |
|
| 233 |
# Vision (MTP does not work with images — omit --spec-type mtp)
|
|
|
|
| 270 |
The model predicts 5 extra tokens per step using its own MTP heads, then verifies them in one pass. No extra model needed.
|
| 271 |
|
| 272 |
```bash
|
| 273 |
+
--spec-type mtp --spec-draft-n-max 5 -np 1
|
| 274 |
```
|
| 275 |
|
| 276 |
+
MTP currently requires `-np 1` (single-sequence mode). Without it, you'll get: `MTP currently supports only n_parallel=1`.
|
| 277 |
+
|
| 278 |
Tune `--spec-draft-n-max`: **5** for general use, **8–10** for code, **3** for creative writing.
|
| 279 |
|
| 280 |
### Draft model (~2.3x faster)
|