hrsvrn/mamba3-180m-dylinear-finewebedu-10B

mamba3_180m_dylinear_finewebedu_10B

Mamba-3 (SISO) language model, ~186.8M parameters, pretrained on 10B tokens of FineWeb-Edu with the Llama-3.1 tokenizer.

Trained as the baseline run for the normalization-free Mamba-3 research project (ICLR 2026 thesis: replacing BCNorm with element-wise stabilizers). This checkpoint keeps BCNorm + BC bias intact and is the reference against which all DyT / Derf / DyISRU / DySoftSign ablations are compared.

Architecture

	Value
Family	Mamba-3 SISO (Llama-style alternating SSM + SwiGLU MLP)
Layers	12
Model dim	768
State size (d_state)	128
Head dim	64
Expand	2
MLP intermediate	1500
Vocab	128256 (Llama-3.1)
Tied embeddings	yes
Sequence length	2048

Training recipe

	Value
Optimizer	AdamW (β=0.9, 0.95), wd=0.1, grad-clip=1.0
LR schedule	cosine, peak 6e-4 → 6e-5, 1000 warmup steps
Batch (tokens)	524288 (256 sequences × 2048)
Total tokens	10B
Total steps	19,073
Precision	bfloat16

See config.yaml for the full, reproducible config snapshot.

Loading

import torch, yaml
from huggingface_hub import hf_hub_download

cfg = yaml.safe_load(open(hf_hub_download("hrsvrn/mamba3-180m-dylinear-finewebedu-10B", "config.yaml")))
ckpt = torch.load(hf_hub_download("hrsvrn/mamba3-180m-dylinear-finewebedu-10B", "final.pt"), map_location="cpu")
# Reconstruct via pretraining/model.py:build_model_from_config(cfg)