File size: 1,262 Bytes
335d699 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 | ---
license: apache-2.0
language:
- en
base_model: Qwen/Qwen3-0.6B
tags:
- parallax
- qwen3
- attention
- wsd
- 0.6b
- pretraining-checkpoint
---
# Attention 0.6B AdamW-WSD - step 18500
Intermediate checkpoint at optimizer step **18500** (of 20000) of the 0.6B
Qwen3-style softmax-attention baseline used in the Parallax mechanism
analysis.
- **Architecture**: Qwen3-0.6B (28 layers, d_model=1024, 16 Q-heads, 8 KV-heads, GQA, RoPE theta 1e6)
- **Optimizer**: AdamW (lr=3e-4, weight decay=0.1, betas=(0.9, 0.95))
- **Scheduler**: WSD (warmup=0, last 20% linearly decayed)
- **Tokens seen at this step**: ~72705.0M sequences * 4096 tokens = ~290.8B tokens
- **Dataset**: Ultra-FineWeb (English split)
- **Tokenizer**: Qwen3-0.6B (vocab 151936)
- **Total steps in run**: 20000 (~80B tokens)
The full training trajectory (every 500 steps) lives in the [Attention 0.6B
AdamW-WSD training trajectory](https://huggingface.co/collections/YifeiZuo)
collection.
## Loading
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"YifeiZuo/attention-0.6b-adamw-wsd-step18500",
trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained("YifeiZuo/attention-0.6b-adamw-wsd-step18500")
```
|