---
license: apache-2.0
base_model: Qwen/Qwen3-4B-Instruct-2507
library_name: transformers
tags:
- speculative-decoding
- eagle3
- specforge
- sglang
- qwen3
- draft-model
- sharegpt
---

# Qwen3-4B-Instruct-2507 EAGLE3 ShareGPT - Epoch 3, Step 70000

This repository contains the epoch **3** (zero-indexed), global step **70000** snapshot from an online EAGLE3 draft-model training run for [Qwen/Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507).

This is a speculative-decoding draft model. It is intended to be used together with the target model and is not a standalone chat or text-generation model.

Part of the [Qwen3-4B-Instruct-2507 EAGLE3 ShareGPT Checkpoints collection](https://huggingface.co/collections/huluhuluu/qwen3-4b-instruct-2507-eagle3-sharegpt-checkpoints-6a8486fa06b05c4a3c1627d2).

## Checkpoint

| Field | Value |
|---|---|
| Target model | `Qwen/Qwen3-4B-Instruct-2507` |
| Training method | Online EAGLE3 with SpecForge |
| Epoch | `3` (zero-indexed) |
| Global step | `70000` / `231810` |
| Draft parameters | `202,700,416` |
| Draft layers | `1` decoder layer |
| Precision | `bfloat16` |
| Sliding-window size | `256` tokens |

## Training Parameters

The values below were recovered from the saved training state and checkpoint config, rather than inferred only from the directory name.

| Parameter | Value |
|---|---|
| Training data | Cleaned ShareGPT JSONL |
| Raw input records | `92,724` before preprocessing |
| Chat template | `qwen` |
| Maximum sequence length | `2048` |
| Epochs | `10` |
| Total optimizer steps | `231810` |
| Per-device batch size | `1` |
| Data-parallel size | `4` |
| Effective global batch size | `4` |
| Gradient accumulation | `1` |
| Target-model tensor parallel size | `1` |
| Optimizer | AdamW with FP32 master parameters |
| Peak learning rate | `1e-4` |
| LR schedule | Cosine annealing with linear warmup |
| Warmup ratio | `0.015` |
| Weight decay | `0.0` |
| Maximum gradient norm | `0.5` |
| EAGLE3 TTT length | `7` |
| Draft attention backend | `sdpa` |
| Target-model backend | SGLang (`flashinfer`) |
| Draft sliding window | `256` |
| Seed | `0` |
| Dataset preprocessing workers | `64` |
| Dataloader workers | `4` |
| Save interval | `5000` steps |
| Log interval | `50` steps |
| SpecForge source revision | `9fbbde8ab5d6ee69fb0af3701330027b8beca37a` |
| Transformers version recorded in config | `4.57.1` |

The training run did not configure an evaluation dataset or experiment tracker. No validation, acceptance-rate, or throughput metrics are reported for this checkpoint. The latest checkpoint is not necessarily the best checkpoint for every workload; compare checkpoints on representative prompts before deployment.

## Architecture

The draft uses `LlamaForCausalLMEagle3` with:

- hidden size 2560;
- intermediate size 9728;
- 32 attention heads and 8 key/value heads;
- target vocabulary size 151,936 and draft vocabulary size 32,000;
- one decoder layer and causal sliding-window attention of 256 tokens.

## Usage with SGLang

```bash
python3 -m sglang.launch_server \
  --model Qwen/Qwen3-4B-Instruct-2507 \
  --speculative-algorithm EAGLE3 \
  --speculative-draft-model-path huluhuluu/Qwen3-4B-Instruct-2507-EAGLE3-ShareGPT-epoch-3-step-70000 \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4
```

The speculative tree settings above are starting values, not benchmark-derived recommendations for this checkpoint. Tune them for the serving workload and SGLang version.

## Files

- `model.safetensors`: EAGLE3 draft weights.
- `config.json`: draft architecture and sliding-window configuration.
- `training_state.pt`: optimizer, scheduler, epoch, global step, and training arguments for resuming training.

`training_state.pt` uses PyTorch pickle-based serialization and is not required for inference. Only deserialize it in a trusted environment. Prefer `model.safetensors` for inference.

## Limitations

- This checkpoint must be paired with the exact target-model family used for training.
- It was trained on a cleaned local ShareGPT file; no public dataset revision was recorded.
- No benchmark or safety evaluation artifacts were saved with the run.
- Speculative-decoding gains depend on prompt distribution, generation settings, hardware, and serving software.

## Licenses

The weights are published under Apache-2.0, following the target model. The SpecForge training code is MIT licensed. Users are responsible for reviewing the terms of the target model and training data for their use case.