froggeric commited on
Commit
24fc15f
·
verified ·
1 Parent(s): 8a804aa

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +5 -5
README.md CHANGED
@@ -75,7 +75,7 @@ cmake --build build --target llama-cli llama-server
75
 
76
  ```bash
77
  llama-server -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
78
- --spec-type mtp --spec-draft-n-max 5 \
79
  --cache-type-k q4_0 --cache-type-v q4_0 \
80
  -np 1 -c 262144 --temp 0.7 --top-k 20 -ngl 99 --port 8081
81
  ```
@@ -84,7 +84,7 @@ That's it. Three optimizations in one command:
84
 
85
  | Flag | What it does | Impact |
86
  |---|---|---|
87
- | `--spec-type mtp --spec-draft-n-max 5` | Multi-Token Prediction (built into the model) | **2.5x faster** generation |
88
  | `--cache-type-k q4_0 --cache-type-v q4_0` | 4-bit KV cache (instead of 16-bit) | **Quarter the KV memory** |
89
  | `-c 262144` | 262K context window | Full native context on **48 GB Mac** with q4_0 KV |
90
 
@@ -225,7 +225,7 @@ curl http://localhost:8081/v1/messages \
225
  ```bash
226
  # Text generation
227
  llama-cli -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
228
- --spec-type mtp --spec-draft-n-max 5 \
229
  --cache-type-k q8_0 --cache-type-v q8_0 \
230
  -np 1 -c 4096 -n 2048 --temp 0.7 -ngl 99 \
231
  -p "Your prompt here"
@@ -270,12 +270,12 @@ Effect on hardware requirements (Q5_K_M, 80K context):
270
  The model predicts 5 extra tokens per step using its own MTP heads, then verifies them in one pass. No extra model needed.
271
 
272
  ```bash
273
- --spec-type mtp --spec-draft-n-max 5 -np 1
274
  ```
275
 
276
  MTP currently requires `-np 1` (single-sequence mode). Without it, you'll get: `MTP currently supports only n_parallel=1`.
277
 
278
- Tune `--spec-draft-n-max`: **5** for general use, **810** for code, **3** for creative writing.
279
 
280
  ### Draft model (~2.3x faster)
281
 
 
75
 
76
  ```bash
77
  llama-server -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
78
+ --spec-type mtp --spec-draft-n-max 3 \
79
  --cache-type-k q4_0 --cache-type-v q4_0 \
80
  -np 1 -c 262144 --temp 0.7 --top-k 20 -ngl 99 --port 8081
81
  ```
 
84
 
85
  | Flag | What it does | Impact |
86
  |---|---|---|
87
+ | `--spec-type mtp --spec-draft-n-max 3` | Multi-Token Prediction (built into the model) | **2.5x faster** generation |
88
  | `--cache-type-k q4_0 --cache-type-v q4_0` | 4-bit KV cache (instead of 16-bit) | **Quarter the KV memory** |
89
  | `-c 262144` | 262K context window | Full native context on **48 GB Mac** with q4_0 KV |
90
 
 
225
  ```bash
226
  # Text generation
227
  llama-cli -m Qwen3.6-27B-Q5_K_M-mtp.gguf \
228
+ --spec-type mtp --spec-draft-n-max 3 \
229
  --cache-type-k q8_0 --cache-type-v q8_0 \
230
  -np 1 -c 4096 -n 2048 --temp 0.7 -ngl 99 \
231
  -p "Your prompt here"
 
270
  The model predicts 5 extra tokens per step using its own MTP heads, then verifies them in one pass. No extra model needed.
271
 
272
  ```bash
273
+ --spec-type mtp --spec-draft-n-max 3 -np 1
274
  ```
275
 
276
  MTP currently requires `-np 1` (single-sequence mode). Without it, you'll get: `MTP currently supports only n_parallel=1`.
277
 
278
+ Tune `--spec-draft-n-max`: **3** is optimal for general use (83% acceptance rate). Values of 12 are more conservative; 4–5 waste compute on rejected tokens.
279
 
280
  ### Draft model (~2.3x faster)
281