DSpark: Inconsistency in num_speculative_tokens
The official DeepSeek command provided for vLLM is: --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"greedy"}'. This indicates it can predict 7 tokens. Why is your recommended value 5?"
The two numbers come from different places: 5 is dspark_block_size in the checkpoint config (the draft head's trained block size, and what most stacks silently infer if you don't set a depth), 7 is what DeepSeek's vLLM recipe passes explicitly. They're not equivalent in practice. On our hardware (4x RTX PRO 6000, SM120, sglang) depth 5 is the one value that corrupts output under real load: depths 3, 4, 6 and 7 all measured clean across ~22K requests, 5 produced repetition loops and garbled text in every screen. Details and repro: https://github.com/sgl-project/sglang/issues/33800. One extra trap for vLLM users: v0.26.0 floors num_speculative_tokens at the checkpoint's block size (vllm/config/speculative.py), so you can't go below 5 there — 6 or 7 are the escapes. Whatever stack you run, set the depth explicitly rather than trusting the default.
Update on my earlier comment, because the story changed: we root-caused the depth-5 corruption, and it turns out it was never the model or the draft head. It was an allocation bug in one sglang code path on SM120 (large einsum transients placed inside an NCCL symmetric-memory region; depth 5's batch shape just happened to trigger the collision). Details and the bisect in sglang#33800.
What that means for GGUF users here: llama.cpp shares none of the affected code, so no depth is dangerous on this stack, and the reports of clean GGUF output all along were exactly right. Pick your draft depth purely on economics: each extra draft position costs a verify pass that only pays if accepted, acceptance is content-dependent (code and tool output accept far more than reasoning prose), and the cost of unaccepted positions is high when your experts live in system RAM. That's why llama.cpp's default of 3 works well on offloaded setups, and why the 2-card GPU-resident configs in these threads profit from more. The checkpoint's dspark_block_size=5 is the head's training block size, not a recommendation — DeepSeek's own recipes pass 7 for datacenter serving.
acceptance is content-dependent (code and tool output accept far more than reasoning prose)
OK, so I am not crazy. I see token generation around during reasoning 7t/s, then it starts writing a code block and jumps up to 11-12t/s. I didn't realize the draft acceptance is dependent on content. Thank you!