---
library_name: speculators
base_model:
- qwen3/qwen3-30b-a3b-instruct-2507
license: apache-2.0
tags:
- speculative-decoding
- dflash
- speculators
---
# RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.dflash
This is a DFlash speculator model for [Qwen/Qwen3-30B-A3B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-30B-A3B-Instruct-2507).
## Training Details
This model was trained using the [Speculators](https://github.com/vllm-project/speculators) library on a subset of [Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered](https://huggingface.co/datasets/Magpie-Align/Magpie-Llama-3.1-Pro-300K-Filtered) and the `train_sft` split of [HuggingFaceH4/ultrachat_200k](https://huggingface.co/datasets/HuggingFaceH4/ultrachat_200k). Responses were regenerated by [Qwen/Qwen3-235B-A22B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507). and stored at [Dataset-Qwen3-235B-Instruct](https://huggingface.co/datasets/inference-optimization/Dataset-Qwen3-235B-Instruct/tree/main)
Commands
Using the [Speculators](https://github.com/vllm-project/speculators) library and the helper scripts provided in the repo.
### Prepare data
```bash
# In virtual environment with speculators installed
python scripts/prepare_data.py \
--model Qwen/Qwen3-30B-A3B-Instruct-2507
--data ./regenerated_data.jsonl \
--assistant-pattern "<\|im_start\|>assistant\s*([\s\S]*?)<\|im_end\|>" \
--output ./output \
--seq-length 16384
```
### Launch vLLM
```bash
# In (separate) virtual environment with vllm installed
CUDA_VISIBLE_DEVICES=0,1 vllm_venv/bin/python scripts/launch_vllm.py \
Qwen/Qwen3-30B-A3B-Instruct-2507 \
--target-layer-ids 1 12 23 34 45 \
--max-model-len 32768 \
--max-num-batched-tokens 32768\
--tensor-parallel-size 2 \
--no-enable-chunked-prefill
```
### Launch training
Must be run once vLLM has finished launching and is running in the background.
```bash
# In virtual environment with speculators installed
CUDA_VISIBLE_DEVICES=2,3 torchrun \
--standalone \
--nproc_per_node 2 \
scripts/train.py \
--verifier-name-or-path Qwen/Qwen3-30B-A3B-Instruct-2507 \
--data-path ./output \
--on-missing generate \
--on-generate delete \
--scheduler-type cosine \
--draft-vocab-size 32000 \
--max-anchors 1024 \
--target-layer-ids 1 12 23 34 45 \
--speculator-type dflash \
--num-layers 5 \
--logger trackio \
--lr 0.0006 \
--epochs 5 \
--sliding-window 2048 \
--sliding-window-indices 0 1 2 3 4 \
--draft-hidden-act silu
```
## Model Specifications
| | |
|---|---|
| **Base Model** | Qwen/Qwen3-30B-A3B-Instruct-2507 |
| **Chat Template** | Qwen/Qwen3-30B-A3B-Instruct-2507 (use `/chat/completions` endpoint) |
| **Format** | Safetensors |
| **License** | Apache 2.0 |
| **Validation Hardware** | Nvidia A100 |
## Deployment
```bash
# Install vLLM from the required PR
pip install git+https://github.com/vllm-project/vllm.git
# Deploy with speculative decoding
vllm serve Qwen/Qwen3-30B-A3B-Instruct-2507 \
--tensor-parallel-size 2 \
--max-num-batched-tokens 32768 \
--attention-backend FLASH_ATTN \
--speculative-config '{
"model": "RedHatAI/Qwen3-30B-A3B-Instruct-2507-speculator.dflash",
"num_speculative_tokens": 15,
"method": "dflash"
}'
```
## Preliminary Evaluations
Per-position token acceptance rates across datasets:
(with reasoning enabled)
| Dataset | Pos 0 | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 | Pos 6 | Avg. Length |
|---------|-------|-------|-------|-------|-------|-------|-------|-------------|
| HumanEval | 83.2% | 66.0% | 51.6% | 40.2% | 31.3% | 24.3% | 18.1% | 4.15 |
| math_reasoning | 87.1% | 72.5% | 59.0% | 47.6% | 37.1% | 28.6% | 21.3% | 4.53 |
| qa | 58.5% | 30.4% | 14.7% | 7.1% | 3.3% | 1.6% | 0.8% | 2.16 |
| question | 67.0% | 40.3% | 24.6% | 15.9% | 10.7% | 7.3% | 5.0% | 2.71 |
| rag | 64.8% | 37.1% | 20.5% | 11.0% | 5.6% | 2.7% | 1.3% | 2.43 |
| summarization | 60.6% | 30.9% | 15.0% | 7.4% | 3.4% | 1.5% | 0.6% | 2.19 |
| tool_call | 65.7% | 42.0% | 27.9% | 19.6% | 12.8% | 8.3% | 5.3% | 2.82 |
| translation | 70.2% | 36.3% | 15.6% | 6.3% | 2.4% | 0.9% | 0.4% | 2.32 |
| writing | 67.0% | 40.3% | 24.6% | 15.9% | 10.7% | 7.3% | 5.0% | 2.71 |
### Latency Speedup
Speedup comparisons of DFlash speculative decoding vs. baseline (no speculation) at varying request rates on Nvidia A100:
## References
**Paper**: [DFlash: Block Diffusion for Flash Speculative Decoding](https://arxiv.org/abs/2602.06036)