| --- |
| license: other |
| language: |
| - en |
| - zh |
| library_name: transformers |
| pipeline_tag: text-generation |
| tags: |
| - specforge |
| - eagle3 |
| - speculative-decoding |
| - draft-model |
| - qwen3 |
| - sharegpt |
| - sliding-window |
| base_model: |
| - Qwen/Qwen3-1.7B |
| --- |
| |
| # Qwen3 1.7B EAGLE3 ShareGPT SW768 - Epoch 9, Step 465000 |
|
|
| This repository contains a SpecForge EAGLE3 draft-model checkpoint for use with `Qwen/Qwen3-1.7B`. |
| It is a draft model for speculative decoding, not a standalone target language model. |
|
|
| ## Checkpoint |
|
|
| | Field | Value | |
| |---|---| |
| | Source run | `qwen3-1.7b-eagle3-sharegpt-sw768` | |
| | Checkpoint | `epoch_9_step_465000` | |
| | Epoch | `9` | |
| | Global step | `465000` | |
| | Files | `config.json`, `model.safetensors`, `training_state.pt` | |
|
|
| ## Training Parameters |
|
|
| | Parameter | Value | |
| |---|---| |
| | Base model | `Qwen/Qwen3-1.7B` | |
| | Method | `SpecForge EAGLE3 online training` | |
| | Framework revision | `9fbbde8ab5d6ee69fb0af3701330027b8beca37a` | |
| | Training data | `sharegpt_train.jsonl` | |
| | Learning rate | `0.0001` | |
| | Batch size | `1` | |
| | Epochs configured | `10` | |
| | Total scheduled steps | `467800` | |
| | Max length | `2048` | |
| | Warmup ratio | `0.015` | |
| | Max grad norm | `0.5` | |
| | TTT length | `7` | |
| | Draft accumulation steps | `1` | |
| | Draft sliding window | `768` | |
| | Save / eval interval | `5000 / 5000` | |
| | Seed | `0` | |
| | TP / DP size | `1 / 2` | |
| | Attention backend | `sdpa` | |
| | Target model backend | `sglang` | |
| | SGLang attention backend | `flashinfer` | |
| | Dataset build workers | `16` | |
|
|
| ## Draft Model Configuration |
|
|
| | Field | Value | |
| |---|---| |
| | Architecture | `LlamaForCausalLMEagle3` | |
| | dtype | `bfloat16` | |
| | Hidden size | `2048` | |
| | Intermediate size | `6144` | |
| | Draft layers | `1` | |
| | Attention heads | `16` | |
| | KV heads | `8` | |
| | Draft vocab size | `32000` | |
| | Vocab size | `151936` | |
| | Max position embeddings | `40960` | |
| | Sliding window | `768` | |
| | Max window layers | `28` | |
|
|
| ## Notes |
|
|
| - The checkpoint weights exclude the frozen target embedding weights; SpecForge loads them from the target model. |
| - The checkpoint is intended to be loaded by SpecForge/EAGLE3-compatible code. |
| - `training_state.pt` is included for provenance and training-state inspection. |
| - No benchmark claim is made in this card. |
|
|
