huluhuluu/qwen3-1.7b-eagle3-sharegpt-sw1024

Qwen3-1.7B EAGLE3 ShareGPT SW1024 Checkpoints

This repository contains all 47 checkpoints from a SpecForge EAGLE3 draft-model training run for the Qwen/Qwen3-1.7B target model. The draft attention window was fixed to 1024 tokens. Checkpoints were saved every 5,000 steps and once at the final step, 231,810.

This is a speculative-decoding draft model, not a standalone language model. Use it with the matching Qwen3-1.7B target model and SpecForge-compatible EAGLE3 inference code.

Checkpoint Layout

Each epoch_<epoch>_step_<step>/ directory contains:

• model.safetensors: BF16 EAGLE3 draft weights
• config.json: draft-model architecture and sliding-window configuration
• training_state.pt: epoch, global step, scheduler state, optimizer metadata, and the original training arguments

Available steps are 5,000 through 230,000 at 5,000-step intervals, plus the final checkpoint at step 231,810. The recommended checkpoint is:

epoch_9_step_231810

Training Configuration

Parameter	Value
Target model	Qwen/Qwen3-1.7B
Method	SpecForge EAGLE3 online training
Training data	ShareGPT-style conversations (sharegpt_train_clean.jsonl)
Chat template	qwen
Epochs	10
Total steps	231,810
Batch size	1 per data-parallel worker
Data parallel size	4
Tensor parallel size	1
Initial learning rate	1e-4
Scheduler	Linear warmup followed by cosine annealing
Warmup ratio	0.015
Max gradient norm	0.5
Precision	BF16 model parameters with FP32 optimizer parameters
Max sequence length	2,048
Draft sliding window	1,024
TTT length	7
Draft accumulation steps	1
Save interval	5,000 steps
Eval interval	5,000 steps (no eval dataset configured)
Draft attention backend	sdpa
Target model backend	sglang
SGLang attention backend	flashinfer
SGLang static memory fraction	0.4
Dataset build processes	16
Dataloader workers	4
Random seed	0
Experiment reporting	Disabled (report_to=none)

The run used SpecForge commit 9fbbde8.

Resume History and Optimizer-State Limitation

Training was stopped after checkpoint step 15,000 and resumed on four GPUs from that checkpoint. The model weights, global step, epoch, and learning-rate scheduler were restored. Training then completed normally at step 231,810.

The FSDP checkpoint writer saved the rank-0 AdamW state without gathering the optimizer shards. Consequently, optimizer_state_dict["state"] is empty in all 47 training_state.pt files. These checkpoints can restore model weights, epoch, global step, and scheduler metadata, but they cannot restore Adam first- and second-moment tensors. A resumed training job will rebuild those moments from zero.

Download

Download the final checkpoint with the Hugging Face CLI:

hf download huluhuluu/qwen3-1.7b-eagle3-sharegpt-sw1024 \
  --include "epoch_9_step_231810/*" \
  --local-dir qwen3-1.7b-eagle3-sharegpt-sw1024

License

The model is released under the Apache 2.0 license, matching the Qwen3 base model license. Users remain responsible for complying with the target model, dataset, and SpecForge licenses and terms.