---
license: other
language:
  - en
  - zh
library_name: transformers
pipeline_tag: text-generation
tags:
  - specforge
  - eagle3
  - speculative-decoding
  - draft-model
  - qwen3
  - sharegpt
  - sliding-window
  - mha
base_model:
  - Qwen/Qwen3-1.7B
---

# Qwen3 1.7B EAGLE3 K2 SW256 MHA ShareGPT

This repository contains a SpecForge EAGLE3 draft-model checkpoint for use with `Qwen/Qwen3-1.7B`.
It is a draft model for speculative decoding, not a standalone target language model.

## Checkpoint

| Field | Value |
|---|---|
| Source run | `qwen3-1.7b-eagle3-k2-sw256-mha-sharegpt` |
| Checkpoint | `epoch_9_step_171560` |
| Epoch | `9` |
| Global step | `171560` |
| Files | `config.json`, `model.safetensors`, `training_state.pt` (when present) |

## Training Parameters

| Parameter | Value |
|---|---|
| Base model | `Qwen/Qwen3-1.7B` |
| Method | `SpecForge EAGLE3 online training` |
| Training data | `sharegpt_train.jsonl` |
| Learning rate | `0.0001` |
| Batch size | `1` |
| Target batch size | `1` |
| Epochs configured | `10` |
| Max length | `2048` |
| Warmup ratio | `0.015` |
| Max grad norm | `0.5` |
| TTT length | `5` |
| Draft layers | `2` |
| Draft sliding window | `256` |
| Save interval | `5000` |
| Eval interval | `5000` |
| Seed | `0` |
| TP / DP size | `1 / 4` |
| Attention backend | `sdpa` |
| Target model backend | `sglang` |
| SGLang attention backend | `flashinfer` |
| Dataset build workers | `64` |

## Draft Model Configuration

| Field | Value |
|---|---|
| Architecture | `LlamaForCausalLMEagle3` |
| dtype | `bfloat16` |
| Hidden size | `2048` |
| Intermediate size | `6144` |
| Draft layers | `2` |
| Attention heads | `16` |
| KV heads | `16` |
| Draft vocab size | `32000` |
| Vocab size | `151936` |
| Max position embeddings | `40960` |
| Sliding window | `256` |
| Max window layers | `2` |
| Future hidden | `Not recorded` |

## Notes

- This is the highest-step local checkpoint available when this repository was published.
- The checkpoint is intended to be loaded by SpecForge/EAGLE3-compatible code.
- `training_state.pt` is included when available for provenance and training-state inspection.
- No benchmark claim is made in this card.