froggeric commited on
Commit
78a27a0
·
verified ·
1 Parent(s): c12be7c

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +5 -3
README.md CHANGED
@@ -77,7 +77,7 @@ cmake --build build --target llama-cli llama-server
77
  llama-server -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
78
  --spec-type mtp --spec-draft-n-max 5 \
79
  --cache-type-k q4_0 --cache-type-v q4_0 \
80
- -c 262144 --temp 0.7 --top-k 20 -ngl 99 --port 8081
81
  ```
82
 
83
  That's it. Three optimizations in one command:
@@ -227,7 +227,7 @@ curl http://localhost:8081/v1/messages \
227
  llama-cli -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
228
  --spec-type mtp --spec-draft-n-max 5 \
229
  --cache-type-k q8_0 --cache-type-v q8_0 \
230
- -c 4096 -n 2048 --temp 0.7 -ngl 99 \
231
  -p "Your prompt here"
232
 
233
  # Vision (MTP does not work with images — omit --spec-type mtp)
@@ -270,9 +270,11 @@ Effect on hardware requirements (Q5_K_M, 80K context):
270
  The model predicts 5 extra tokens per step using its own MTP heads, then verifies them in one pass. No extra model needed.
271
 
272
  ```bash
273
- --spec-type mtp --spec-draft-n-max 5
274
  ```
275
 
 
 
276
  Tune `--spec-draft-n-max`: **5** for general use, **8–10** for code, **3** for creative writing.
277
 
278
  ### Draft model (~2.3x faster)
 
77
  llama-server -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
78
  --spec-type mtp --spec-draft-n-max 5 \
79
  --cache-type-k q4_0 --cache-type-v q4_0 \
80
+ -np 1 -c 262144 --temp 0.7 --top-k 20 -ngl 99 --port 8081
81
  ```
82
 
83
  That's it. Three optimizations in one command:
 
227
  llama-cli -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
228
  --spec-type mtp --spec-draft-n-max 5 \
229
  --cache-type-k q8_0 --cache-type-v q8_0 \
230
+ -np 1 -c 4096 -n 2048 --temp 0.7 -ngl 99 \
231
  -p "Your prompt here"
232
 
233
  # Vision (MTP does not work with images — omit --spec-type mtp)
 
270
  The model predicts 5 extra tokens per step using its own MTP heads, then verifies them in one pass. No extra model needed.
271
 
272
  ```bash
273
+ --spec-type mtp --spec-draft-n-max 5 -np 1
274
  ```
275
 
276
+ MTP currently requires `-np 1` (single-sequence mode). Without it, you'll get: `MTP currently supports only n_parallel=1`.
277
+
278
  Tune `--spec-draft-n-max`: **5** for general use, **8–10** for code, **3** for creative writing.
279
 
280
  ### Draft model (~2.3x faster)