unsloth

/

DeepSeek-V4-Flash-0731-GGUF

like 759

Follow

Unsloth AI 34.7k

I wish this was more obvious than buried, but using llama.cpp its something like
-hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q8_K_XL --temp 1.0 --top-p 0.95 --spec-type draft-dspark --spec-draft-n-max 4
I don't know the sweet spot on spec-draft-n-max
recipes.vllm.ai has the stuff for vllm configs for spec decoding.

oh good the docs has it spelled out! using --spec-draft-n-max 3
https://unsloth.ai/docs/models/deepseek-v4#dspark-speculative-decoding

Getting an impression that DSpark is useless with a 4-channel DDR4 + 2x 3090 rig (PCIE 4.0 x16, no nvlink; Win11 + Unsloth Studio), since there's no benefit whatsoever.

1x 3090 no DSpark: ~10 t/s generation --- Q8K_XL fits with 100K context (44 GPU layers, 43 MOE CPU layers), using VRAM quite sparingly and allowing for a higher batch size (2K to 8K).
2x 3090 with or without DSpark, no difference: ~7 t/s generation --- interestingly, it consumes the available VRAM pool unevenly (23GB on one GPU, and about a half of that on another GPU). EDIT 1: this is no longer the case after Unsloth Studio update (both GPUs now show almost equal VRAM usage), however, generation speed remains inferior.
EDIT 2: "prefer maximum performance" in NVIDIA app got me to 9 - 10 t/s with DSpark and 2x 3090, which is still weird.

@danielhanchen Any chances we could get dspark quants where the routed experts are quanted down to save on memory on the dspark model?

EDIT 2: "prefer maximum performance" in NVIDIA app got me to 9 - 10 t/s with DSpark and 2x 3090, which is still weird.

Same 2x3090 + 128GB DRAM @ 4800 (OC, 4-sticks) 8.8-9.5 at full quant. Happy as a clam. Model chugs while i do other things.

No amount of faffing got me better results than simple -fit on with llama.cpp.

9995WX, 1024GB DDR5 6400MT/s ECC, RTX Pro 6000 w/96GB of GDDR7.

I did this series top down, then realized I should collect ram usage so i did them in reverse bottom up. A lot of numbers were the same or nearly identical so I only have one set, but as I moved up the list, the things that were slow became faster and I'm not sure why. The llama.cpp webUI consists of one prompt and I kept regenerating the one answer. On one attempt I started a new chat with same question and achieved basically identical performance as the regenerated answer.

I'm sitting around 166w peak on my GPU. With llama-b10509-bin-win-cuda-13.3-x64\llama-server.exe (and some of my usual settings) and
--ctx-size 262144 --load-mode dio -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q8_K_XL --temp 1.0 --top-p 0.95 --spec-type draft-dspark --spec-draft-n-max 3 --reasoning on --fit on
for my sample prompt I get
1st, 821 tokens, 1min 27s, 9.34 t/s, (didn't capture ram usage first run)
2nd, 829 tokens, 38s, 21.54 t/s, 108 GB total system RAM after prompt (+ 95.8 GB of GPU VRAM holding part of model).
1st, draft acceptance = 0.73152 ( 564 accepted / 771 generated), mean len = 3.19
2nd, draft acceptance = 0.75033 ( 574 accepted / 765 generated), mean len = 3.25

--ctx-size 262144 --load-mode dio -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q8_K_XL --temp 1.0 --top-p 0.95 --spec-type draft-dspark --spec-draft-n-max 3 --reasoning on
1st, 1,167 tokens, 2min 3s, 9.44 t/s, (didn't capture ram usage first run)
2nd, 1,020 tokens, 44s, 22.86 t/s, 108 GB total system RAM after prompt (+ 95.8 GB of GPU VRAM holding part of model).
1st, draft acceptance = 0.75936 ( 811 accepted / 1068 generated), mean len = 3.28
2nd, draft acceptance = 0.78878 ( 717 accepted / 909 generated), mean len = 3.37

--ctx-size 262144 --load-mode dio -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q8_K_XL --temp 1.0 --top-p 0.95 --spec-type draft-dspark --spec-draft-n-max 3 --reasoning on --seed 3407
1st, 792 tokens, 1min 21s, 9.72 t/s, (didn't capture ram usage first run)
2nd, 792 tokens, 35s, 22.59 t/s, (When I re-ran to get system ram usage I got these speeds) (108 GB total system RAM after prompt (+ 95.8 GB of GPU VRAM holding part of model).
1st, draft acceptance = 0.77451 ( 553 accepted / 714 generated), mean len = 3.32

--ctx-size 262144 --load-mode dio -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q8_K_XL --temp 1.0 --top-p 0.95 --spec-type draft-dspark --spec-draft-n-max 3 --reasoning on --seed 3407 --cpu-moe
936 tokens, 22s, 41.54 t/s, 170 GB total system RAM after prompt (+ 34.6 GB of GPU VRAM holding part of model).
draft acceptance = 0.78095 ( 656 accepted / 840 generated), mean len = 3.34

--ctx-size 262144 --load-mode dio -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q8_K_XL --temp 1.0 --top-p 0.95 --spec-type draft-dspark --spec-draft-n-max 3 --reasoning on --seed 3407 --cpu-moe -ctk q8_0 -ctv q8_0
916 tokens, 22s, 40.63 t/s, 170 GB total system RAM after prompt (+ 31.4 GB of GPU VRAM holding part of model).
draft acceptance = 0.76499 ( 638 accepted / 834 generated), mean len = 3.29

--ctx-size 262144 --load-mode dio -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q8_K_XL --temp 1.0 --top-p 0.95 --spec-type draft-dspark --spec-draft-n-max 3 --reasoning on --seed 3407 --cpu-moe -ctk q8_0 -ctv q8_0 --fit on
916 tokens, 22s, 40.61 t/s, 170 GB total system RAM after prompt (+ 31.4 GB of GPU VRAM holding part of model).
draft acceptance = 0.76499 ( 638 accepted / 834 generated), mean len = 3.29

--ctx-size 262144 --load-mode dio -hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q8_K_XL --temp 1.0 --top-p 0.95 --spec-type draft-dspark --spec-draft-n-max 3 --reasoning on --seed 3407 --fit on
792 tokens, 36s, 21.91 t/s, 108 GB total system RAM after prompt (+ 95.8 GB of GPU VRAM holding part of model).
draft acceptance = 0.77451 ( 553 accepted / 714 generated), mean len = 3.32

Using real long prompts the ones that were doing ~40t/s seem to stabilize out around 30-40 t/s
Maybe this is helpful for someone.