decoding with DSpark is slower on llama
System:
- Nvidia RTX 5090 (32GB VRAM) with drivers version 610 (CUDA 13.3)
- Intel Xeon W5-3425 12cores (using only 10)
- 512GB RAM (8 channels * 64GB DDR5-4800)
- Ubuntu 24
Without DSpark:
ctx-size=131702
batch-size=8192
ubatch-size=8192
getting 11.1 t/s
srv load_model: initializing, n_slots = 4, n_ctx_slot = 131840, kv_unified = 'true'
...
prompt eval time = 15433.65 ms / 2447 tokens ( 158.55 tokens per second)
eval time = 52628.98 ms / 587 tokens ( 11.15 tokens per second)
total time = 68062.63 ms / 3034 tokens
graphs reused = 578
DSpark with spec-draft-n-max=3:
ctx-size=131702
batch-size=8192
ubatch-size=8192
spec-type=draft-dspark
spec-draft-model=/root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/57326b941c4603e24d1a5e71c22520c66e086eb8/dspark/dspark-DeepSeek-V4-Flash-0731-BF16.gguf
fit-target=16000
getting ~5.5 t/s:
srv load_model: initializing, n_slots = 4, n_ctx_slot = 131840, kv_unified = 'true'
common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dspark'
common_speculative_impl_draft_dflash: - n_max=3, n_min=0, p_min=0.00
common_speculative_impl_draft_dflash: - block_size=5, mask_token_id=128799, n_extract=3
...
prompt eval time = 18349.27 ms / 2447 tokens ( 133.36 tokens per second)
eval time = 113463.03 ms / 628 tokens ( 5.53 tokens per second)
total time = 131812.30 ms / 3075 tokens
graphs reused = 1485
draft acceptance = 0.52812 ( 385 accepted / 729 generated), mean len = 2.58
DSpark with spec-draft-n-max=1:
ctx-size=131702
batch-size=8192
ubatch-size=8192
spec-type=draft-dspark
spec-draft-model=/root/.cache/huggingface/hub/models--unsloth--DeepSeek-V4-Flash-0731-GGUF/snapshots/57326b941c4603e24d1a5e71c22520c66e086eb8/dspark/dspark-DeepSeek-V4-Flash-0731-BF16.gguf
spec-draft-n-max=1
fit-target=16000
getting ~5.8 t/s:
srv load_model: initializing, n_slots = 4, n_ctx_slot = 131840, kv_unified = 'true'
common_speculative_impl_draft_dflash: adding speculative implementation 'draft-dspark'
common_speculative_impl_draft_dflash: - n_max=1, n_min=0, p_min=0.00
common_speculative_impl_draft_dflash: - block_size=5, mask_token_id=128799, n_extract=3
...
prompt eval time = 19037.97 ms / 2447 tokens ( 128.53 tokens per second)
eval time = 178361.43 ms / 1049 tokens ( 5.88 tokens per second)
total time = 197399.40 ms / 3496 tokens
graphs reused = 575
draft acceptance = 0.77027 ( 456 accepted / 592 generated), mean len = 1.77
my docker-compose.yaml (same for all configs above)
services:
llama-router:
image: ghcr.io/ggml-org/llama.cpp:server-cuda13-b10257
container_name: router
devices:
- "nvidia.com/gpu=all"
ports:
- "8080:8080"
environment:
- NVIDIA_VISIBLE_DEVICES=all
- NVIDIA_DRIVER_CAPABILITIES=compute,utility
volumes:
- /var/lib/docker/.cache:/root/.cache:ro
- ./models.ini:/app/models.ini:ro
entrypoint: ["./llama-server"]
command: >
--models-max 1
--models-preset ./models.ini
--host 0.0.0.0 --port 8080
--offline
--tools all
A few things to try:
--spec-draft-p-min at various values around 0.8
--cpu-moe --spec-draft-cpu-moe
--n-cpu-moe tuning and --spec-draft-n-cpu-moe tuning (general idea is to set them as low as possible without crashing)
No --spec-draft-cpu-moe, but a quantized drafter than doesn't eat up all your vram
The MTP from the preview version of deepseek v4 flash, instead of dspark from 0731.
(Disclaimer: I never managed to actually get it running faster with the draft heads than without, but I'm starting to think that's expected on a DDR4 laptop with a Turing GPU...)
That result is consistent with what we measured: speculative decoding only pays when the draft's acceptance beats the verify cost, and acceptance for this head is strongly content-dependent. On our rig it accepts ~2.2 of a 5-token window on reasoning-heavy prose but 3.4-4.0 on code and tool-call output. If your workload is chat/reasoning, a deep window costs more verify compute than it returns, and on a single 5090 with the experts offloaded that verify pass is expensive. Two things to check in your logs: the per-position acceptance rate (if position 2+ is mostly rejected, depth is wasted), and whether throughput recovers at depth 1-2. Also worth ruling out the depth-5 trap: the checkpoint's default draft depth measurably corrupts on SM120 in our testing (sglang#33800), and degraded-but-not-garbled output sometimes reads as "just slower".