Add training-parameter model card
Browse files
README.md
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
---
|
| 2 |
+
license: apache-2.0
|
| 3 |
+
base_model: Qwen/Qwen3-4B-Instruct-2507
|
| 4 |
+
library_name: transformers
|
| 5 |
+
tags:
|
| 6 |
+
- speculative-decoding
|
| 7 |
+
- eagle3
|
| 8 |
+
- specforge
|
| 9 |
+
- sglang
|
| 10 |
+
- qwen3
|
| 11 |
+
- draft-model
|
| 12 |
+
- sharegpt
|
| 13 |
+
- sliding-window-512
|
| 14 |
+
---
|
| 15 |
+
|
| 16 |
+
# Qwen3-4B-Instruct-2507 EAGLE3 ShareGPT (SW512)
|
| 17 |
+
|
| 18 |
+
Online EAGLE3 draft-model training run with SpecForge and a draft sliding window of `512` tokens. This archive contains 47 checkpoints from `epoch_0_step_5000` through `epoch_9_step_231810`; each checkpoint is published as a separate Hub model repository in the companion collection.
|
| 19 |
+
|
| 20 |
+
This is a speculative-decoding draft model, not a standalone chat model. Pair it with the exact target model family.
|
| 21 |
+
|
| 22 |
+
## Training parameters
|
| 23 |
+
|
| 24 |
+
| Parameter | Value |
|
| 25 |
+
|---|---|
|
| 26 |
+
| Target model | `Qwen/Qwen3-4B-Instruct-2507` |
|
| 27 |
+
| Training method | Online EAGLE3 / SpecForge |
|
| 28 |
+
| Training data | Cleaned ShareGPT JSONL (local source; revision not recorded) |
|
| 29 |
+
| Epochs | `10` |
|
| 30 |
+
| Total optimizer steps | `231810` |
|
| 31 |
+
| Per-device batch size | `1` |
|
| 32 |
+
| Data-parallel size | `4` |
|
| 33 |
+
| Effective global batch size | `4` |
|
| 34 |
+
| Gradient accumulation | `1` |
|
| 35 |
+
| Learning rate | `1e-4` |
|
| 36 |
+
| LR schedule | Linear warmup (`1.5%`) then cosine annealing |
|
| 37 |
+
| Weight decay | `0.0` |
|
| 38 |
+
| Max gradient norm | `0.5` |
|
| 39 |
+
| Maximum sequence length | `2048` |
|
| 40 |
+
| EAGLE3 TTT length | `7` |
|
| 41 |
+
| Draft sliding window | `512` tokens |
|
| 42 |
+
| Draft attention | `sdpa` |
|
| 43 |
+
| Target backend | SGLang (`flashinfer`) |
|
| 44 |
+
| Tensor parallel size | `1` |
|
| 45 |
+
| Seed | `0` |
|
| 46 |
+
| Save / log interval | `5000` / `50` steps |
|
| 47 |
+
| Dataset / dataloader workers | `64` / `4` |
|
| 48 |
+
|
| 49 |
+
## Architecture
|
| 50 |
+
|
| 51 |
+
`LlamaForCausalLMEagle3`, one decoder layer, hidden size `2560`, intermediate size `9728`, 32 attention heads, 8 key/value heads, draft vocabulary size `32000`, target vocabulary size `151936`, `bfloat16` weights, and causal sliding-window attention of 512 tokens.
|
| 52 |
+
|
| 53 |
+
## Checkpoint files
|
| 54 |
+
|
| 55 |
+
Every checkpoint repository contains `model.safetensors`, `config.json`, and `training_state.pt`. The latter stores optimizer/scheduler state and training arguments for resuming and should only be deserialized in a trusted environment. Prefer `model.safetensors` for inference.
|
| 56 |
+
|
| 57 |
+
## Usage
|
| 58 |
+
|
| 59 |
+
Use a checkpoint repository as the SGLang speculative draft path with `Qwen/Qwen3-4B-Instruct-2507` and the EAGLE3 speculative-decoding settings supported by your SGLang version. Tree settings should be benchmarked for the serving workload. No evaluation or safety metrics were recorded for this run.
|