magnitudedev

/

Qwen3.8-27B-DSpark-GGUF

Like 9

Follow

Magnitude 3

RTX 6000 Pro, maximum thinking, Unsloth Q8_K_XL, ~70K total token generation prompt, deterministic, ngram-mod disabled:

no draft: 53

dspark 2 = 100
dspark 3 = 123
dspark 4 = 139
dspark 5 = 140
dspark 6 = 154
dspark 7 = 159
dspark 8 = 158
dspark 9 = 158
dspark 10 = 158

mtp 2 = 105
mtp 3 = 120
mtp 4 = 131
mtp 5 = 132
mtp 6 = 132

can you give the exact llama.cpp command args you used?

The block size is 7 so testing beyond 7 draft tokens is meaningless as the server clamps it to 7.

EDIT: I have made some mistakes regarding the setup. I will report performance once correct testing is done.

can you give the exact llama.cpp command args you used?

I used prediction-tweaked variants on my standard startup script:

set CUDA_VISIBLE_DEVICES=0

A:\AI\llama-nightly\llama-server.exe ^
    -m "A:\AI\Llama\Models\Qwen3.8-27B\Qwen3.8-27B-UD-Q8_K_XL.gguf" ^
    -ngl 999 ^
    -ub 2048 ^
    -dio ^
    -c 524288 ^
    -fa on ^
    -fit off ^
    -ctk f16 ^
    -ctv f16 ^
    --alias local ^
    --no-mmap ^
    --parallel 2 ^
    --spec-type draft-dspark ^
    --model-draft "A:\AI\Llama\Models\Qwen3.8-27B\Qwen3.8-27B-DSpark-Q8_0.gguf" ^
    --spec-draft-n-max 7 ^
    --host 0.0.0.0 ^
    --port 5001 ^
    --jinja ^
    --cache-ram 16384 ^
    --mmproj "A:\AI\Llama\Models\Qwen3.8-27B\mmproj-BF16.gguf" ^
    --image-min-tokens 1024 ^
    --reasoning-preserve

Note ngram-mod disabled to ensure the only thing being tested is MTP/dspark. The actual script I ran had temperature set to 0 and seed fixed at 1234. Parallel is enabled (just part of my template) but not being used, the test was strictly single-user.

The actual test was writing a ~2000 line python script. Total tokens generated per test was north of 70K, IIRC.

The block size is 7 so testing beyond 7 draft tokens is meaningless as the server clamps it to 7.

My own testing result is very different. It shows dspark is a significant downgrade - ranging from -25% to -40% - in throughput vs native MTP on 3 different models (two claude distills, one base qwen).

If you're running multi-GPU that might explain it. You could try forcing llama-server to run the drafter on a single GPU, it might have an effect. I'm running a single 6000 Pro and it has such a disgusting amount of compute that normally goes unused during token generation, maybe that's what's helping in my case. I'm hardly an expert though, just spitballing.

TG speed also increases a fair bit while outputting the actual python code; during chain-of-thought the speed is slower but MTP has a very similar proportional reduction.

Just thought I'd report my results in case it would be of interest to anyone.

DSpark vs ngram vs MTP vs DFlash2 on RTX 5090s — updated for b10851 (branchless MMVQ)

Sharing results on a different hardware profile than the single-GPU numbers above:
GeForce RTX 5090 (32 GB) — originally 3× cards in layer split (GeForce has no P2P —
nvidia-smi topo -p2p r reports GNS, so tensor split is a regression on this setup),
now extended with single-GPU + concurrency tests on build 10851.

Update TL;DR: build 10851 (includes #26705, branchless Q4_K/Q5_K MMVQ) changes the
speculation economics on Blackwell. Draft verification is a small-batch MMVQ call
(ncols_dst = 2–8) — exactly the region #26705 sped up (+13–23% at B=4–6 measured on
this card, batched-bench) — so drafts that used to be marginal now pay, and DFlash2
went from "+9–12% single-stream" to +45%, while no longer losing under concurrent
load. Details in the new section below; the b10558 single-request results are kept
since they still hold for that build/topology.

Part 1 — DSpark vs ngram vs MTP, single request, 3× 5090 layer split (build 10558)

Setup: target Qwen3.8-27B-Q4_K_M.gguf (16 GB), draft = this repo's Q8_0
requantized to Q4_K_M (769 MB, llama-quantize --allow-requantize).
Flags: --flash-attn on --cont-batching --cache-type-k q4_0 --cache-type-v q4_0 --batch-size 8192 --ubatch-size 1024 --spec-type draft-dspark --spec-draft-n-max 5 --cache-type-k-draft q8_0 --cache-type-v-draft q8_0. Undrafted baseline: ~78 t/s.

n-max: 5 beats 7 on this hardware. Contrary to the RTX 6000 Pro sweep above
(peak at 7), n-max 7 was clearly worse here: acceptance collapses (JSON 26% vs 59%
at n-max 5) and the wasted verification compute costs real throughput on a
multi-GPU layer-split pipeline. Worth re-sweeping n-max per hardware.

DSpark vs native MTP (--spec-type draft-mtp, n-max 3):

Split by workload: DSpark wins code/JSON, MTP wins prose by a lot (47% vs 17%
acceptance) and is far lighter on VRAM.

Part 2 — NEW: DFlash2 + concurrency stress, single 5090, build 10851

Re-tested after #26705 landed (branchless Q4_K/Q5_K unpack in MMVQ). The old
"drafts don't pay under load" rule deserved a re-check, because verification
batches are exactly the MMVQ instances that got cheaper.

Setup: single 5090, target Qwen3.8-27B-Q4_K_M.gguf, split KV, 8 slots × 20480,
-fa on -ub 256 -b 8192, KV q4_0, draft KV q8_0, --spec-draft-n-max 5
(--spec-draft-p-min 0.0 for DFlash2). DFlash2 draft: 1.06 GiB Q4_K_M. Fresh server
per config; generic mixed prompts (not the code-heavy set from Part 1 — see
workload caveat below). Load = 1 / 12 / 24 concurrent streams, 300–500 tokens each.

Findings:

1. DFlash2 single-stream went from +9–12% (pre-#26705 builds) to +45% on the same
   card and model — 3-token accepted runs at ~40% acceptance now pay off big because
   the verify batch is cheap. And it no longer loses at saturation: +6% at 12
   streams, noise at 24. On this build, DFlash2 on a single-GPU instance is close to
   a free lunch at every load level.
2. Workload matters more than anything for DSpark. On these generic prompts it
   accepts only 14–16% (vs 87–95% on the code workloads in Part 1) — and at that rate
   the draft cost is pure waste under load: −25–32% with concurrency. If your traffic
   is code/JSON-heavy, Part 1's numbers still apply; for mixed agent text, DSpark
   hurts.
3. Scope caveat: Part 2 is single-GPU. Multi-GPU layer split has a separate
   draft-sync penalty (the draft context syncs every ubatch and breaks pipeline
   overlap, see #27428) that I have NOT retested on b10851 — don't extrapolate the
   DFlash2 win to layer-split instances without your own A/B.
4. Method note: wall-clock aggregate t/s swung ±35% between identical reps
   (my no-draft config read 273 then 171 t/s aggregate at 12 streams) — per-stream
   decode medians are stable and are what's tabled above. Single-run aggregate
   numbers under concurrency will mislead in either direction.

Other notes (Part 1 hardware, still valid)

• Q8_0 vs Q4_K_M draft: no measurable difference (184–235 vs 165–252 t/s on code,
  acceptance overlaps) — the Q4_K_M requant is the better deal.
• The DSpark draft's KV cache scales with total context and must fit on the first
  CUDA device; native MTP's draft context lands on the last one. Skewing
  --tensor-split (e.g. 4,6,6 for DSpark, 6,6,4 for MTP) to unload the
  draft-hosting card buys ~33% more total context.
• Per-request speed swings with draft acceptance (code runs ranged ~130–250 t/s on
  the same prompt) — single-run numbers can mislead in either direction; under
  concurrency, use per-stream medians (see Part 2 method note).