Qwen3-0.6B-DSpark

A DSpark draft model for speculative decoding with Qwen/Qwen3-0.6B as the verifier, trained with speculators. The drafter proposes 4 tokens at a time and the verifier checks them in one forward pass, so output is identical to running the verifier alone — a lossless speedup.

Training code: rasyosef/train-dspark-draft-models.

Trained on 1,600 samples as a pipeline demonstration, not a deployment-ready drafter.

Usage

vLLM loads the verifier automatically from the config — don't pass it separately.

vllm serve yosefw/Qwen3-0.6B-DSpark --port 8000 --gpu-memory-utilization 0.75

Then query the OpenAI-compatible endpoint at http://localhost:8000/v1.

Details

5 Qwen3 layers (hidden size 1024, sliding-window attention), ~0.3B params, bfloat16. Block size 4, draft vocabulary reduced to 32,000, aux hidden-state layers 2/14/25, confidence head with Markov (rank 256).

Trained for 5 epochs at lr 3e-4 on 1,600 Magpie prompts regenerated by the verifier itself, with a {"ce": 0.1, "tv": 0.9} loss. Online mode: a vLLM server exported hidden states per batch during training. 2× T4, 8 h 54 min. speculators 0.8.0.dev197, vLLM 0.27.1.

Evaluation

evaluate.py throughput on 2× T4 across the nine RedHatAI/speculator_benchmarks subsets:

subset	acceptance_length	pos_0	pos_1	pos_2	pos_3
math_reasoning	1.9702	0.2673	0.1351	0.0647	0.0281
HumanEval	1.8709	0.2504	0.1168	0.0501	0.0204
writing	1.8068	0.2074	0.0953	0.0435	0.0190
qa	1.8011	0.2681	0.1258	0.0396	0.0188
question	1.7903	0.2753	0.1265	0.0543	0.0217
tool_call	1.7225	0.2036	0.0868	0.0384	0.0152
rag	1.6370	0.2114	0.0898	0.0390	0.0158
summarization	1.4854	0.1568	0.0569	0.0224	0.0090
translation	1.4356	0.1433	0.0519	0.0182	0.0063

Weighted across all subsets: 1.806 (435,398 accepted tokens over 539,885 drafts). Acceptance is highest where the verifier's next token is most predictable — code and math — and lowest on translation and summarization.

Limitations

Trained on only 1,600 samples, so acceptance would improve considerably with more data. Works only with Qwen/Qwen3-0.6B and is not usable as a standalone model. Real-world speedup depends on your traffic mix, and because verification is lossless, the verifier's own behavior and biases carry through unchanged.

Related

yosefw/Qwen3-0.6B-DSpark-v2 — same setup trained in offline mode on 1,200 samples.

License

Apache-2.0, matching both speculators and the verifier.