license: apache-2.0
base_model: Qwen/Qwen3-4B-Instruct-2507
library_name: transformers
tags:
- speculative-decoding
- eagle3
- specforge
- sglang
- qwen3
- draft-model
- sharegpt
- sliding-window-512
Qwen3-4B-Instruct-2507 EAGLE3 ShareGPT (SW512)
Online EAGLE3 draft-model training run with SpecForge and a draft sliding window of 512 tokens. This archive contains 47 checkpoints from epoch_0_step_5000 through epoch_9_step_231810; each checkpoint is published as a separate Hub model repository in the companion collection.
This is a speculative-decoding draft model, not a standalone chat model. Pair it with the exact target model family.
Training parameters
| Parameter | Value |
|---|---|
| Target model | Qwen/Qwen3-4B-Instruct-2507 |
| Training method | Online EAGLE3 / SpecForge |
| Training data | Cleaned ShareGPT JSONL (local source; revision not recorded) |
| Epochs | 10 |
| Total optimizer steps | 231810 |
| Per-device batch size | 1 |
| Data-parallel size | 4 |
| Effective global batch size | 4 |
| Gradient accumulation | 1 |
| Learning rate | 1e-4 |
| LR schedule | Linear warmup (1.5%) then cosine annealing |
| Weight decay | 0.0 |
| Max gradient norm | 0.5 |
| Maximum sequence length | 2048 |
| EAGLE3 TTT length | 7 |
| Draft sliding window | 512 tokens |
| Draft attention | sdpa |
| Target backend | SGLang (flashinfer) |
| Tensor parallel size | 1 |
| Seed | 0 |
| Save / log interval | 5000 / 50 steps |
| Dataset / dataloader workers | 64 / 4 |
Architecture
LlamaForCausalLMEagle3, one decoder layer, hidden size 2560, intermediate size 9728, 32 attention heads, 8 key/value heads, draft vocabulary size 32000, target vocabulary size 151936, bfloat16 weights, and causal sliding-window attention of 512 tokens.
Checkpoint files
Every checkpoint repository contains model.safetensors, config.json, and training_state.pt. The latter stores optimizer/scheduler state and training arguments for resuming and should only be deserialized in a trusted environment. Prefer model.safetensors for inference.
Usage
Use a checkpoint repository as the SGLang speculative draft path with Qwen/Qwen3-4B-Instruct-2507 and the EAGLE3 speculative-decoding settings supported by your SGLang version. Tree settings should be benchmarked for the serving workload. No evaluation or safety metrics were recorded for this run.