File size: 2,437 Bytes
bf6ca4f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 | ---
license: apache-2.0
base_model: Qwen/Qwen3-4B-Instruct-2507
library_name: transformers
tags:
- speculative-decoding
- eagle3
- specforge
- sglang
- qwen3
- draft-model
- sharegpt
- sliding-window-512
---
# Qwen3-4B-Instruct-2507 EAGLE3 ShareGPT (SW512)
Online EAGLE3 draft-model training run with SpecForge and a draft sliding window of `512` tokens. This archive contains 47 checkpoints from `epoch_0_step_5000` through `epoch_9_step_231810`; each checkpoint is published as a separate Hub model repository in the companion collection.
This is a speculative-decoding draft model, not a standalone chat model. Pair it with the exact target model family.
## Training parameters
| Parameter | Value |
|---|---|
| Target model | `Qwen/Qwen3-4B-Instruct-2507` |
| Training method | Online EAGLE3 / SpecForge |
| Training data | Cleaned ShareGPT JSONL (local source; revision not recorded) |
| Epochs | `10` |
| Total optimizer steps | `231810` |
| Per-device batch size | `1` |
| Data-parallel size | `4` |
| Effective global batch size | `4` |
| Gradient accumulation | `1` |
| Learning rate | `1e-4` |
| LR schedule | Linear warmup (`1.5%`) then cosine annealing |
| Weight decay | `0.0` |
| Max gradient norm | `0.5` |
| Maximum sequence length | `2048` |
| EAGLE3 TTT length | `7` |
| Draft sliding window | `512` tokens |
| Draft attention | `sdpa` |
| Target backend | SGLang (`flashinfer`) |
| Tensor parallel size | `1` |
| Seed | `0` |
| Save / log interval | `5000` / `50` steps |
| Dataset / dataloader workers | `64` / `4` |
## Architecture
`LlamaForCausalLMEagle3`, one decoder layer, hidden size `2560`, intermediate size `9728`, 32 attention heads, 8 key/value heads, draft vocabulary size `32000`, target vocabulary size `151936`, `bfloat16` weights, and causal sliding-window attention of 512 tokens.
## Checkpoint files
Every checkpoint repository contains `model.safetensors`, `config.json`, and `training_state.pt`. The latter stores optimizer/scheduler state and training arguments for resuming and should only be deserialized in a trusted environment. Prefer `model.safetensors` for inference.
## Usage
Use a checkpoint repository as the SGLang speculative draft path with `Qwen/Qwen3-4B-Instruct-2507` and the EAGLE3 speculative-decoding settings supported by your SGLang version. Tree settings should be benchmarked for the serving workload. No evaluation or safety metrics were recorded for this run.
|