README.md

metadata

license: apache-2.0
language:
  - en
tags:
  - text-generation
  - causal-lm
  - pytorch
  - sft
  - instruction-tuned
  - chat
  - hybrid
  - gated-deltanet
  - gqa
  - monostich
pipeline_tag: text-generation
library_name: tiny_gdn
datasets:
  - HuggingFaceTB/smoltalk
  - NousResearch/Hermes-3-Dataset
base_model: kerzgrr/Monostich-2-base
model-index:
  - name: Monostich-2
    results: []

Monostich-2

Instruction-tuned chat model (~150M) — Monostich-2 family

Second-generation Monostich-class model — hybrid GDN-2 + GQA, chat-tuned

---

What this is

Monostich-2 is the supervised fine-tuned (SFT) chat checkpoint for the Monostich-2 family.

• Base (pretrain): kerzgrr/Monostich-2-base
• Successor to kerzgrr/Monostich (~100M LLaMA-style)
• Architecture: hybrid Gated DeltaNet-2 + gated GQA

---

Training

Pretrain → SFT

Stage	Details
Base	FineWeb-Edu pretrain → Monostich-2-base
SFT mix	HuggingFaceTB/smoltalk + NousResearch/Hermes-3-Dataset
Epochs	1 full epoch
Wall time	21.70 hours
Final step	optimizer step 10619
Weights	EMA (Hub model.safetensors is EMA @ bfloat16)
Seq length	8192 (packed SFT)
Peak LR	1 × 10⁻⁴ AdamW, cosine → 10% min
Final val loss (EMA)	1.532 (ppl ≈ 4.63)

Chat template (ChatML)

<|begin_of_text|><|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{assistant}<|im_end|>

Generation prompt ends at <|im_start|>assistant\n.

---

Model Architecture

Same TinyGDN hybrid as the base (~149.1M params):

Layers	32 (GDN-2 ×3 + GQA every 4th)
Hidden	512
MLP	SwiGLU 1472
Attention	4 Q / 1 KV, head dim 128, partial RoPE
Linear	Gated DeltaNet-2, 4 heads × 128
Vocab	49,152 BPE

---

Install & run

pip install torch safetensors tokenizers huggingface_hub
hf download kerzgrr/Monostich-2 inference.py --local-dir .
python inference.py --prompt "What is the capital of France?"

inference.py auto-downloads weights/tokenizer/tiny_gdn/ and auto-installs pinned flash-linear-attention (Windows applies Hub patches). Git required on PATH.

Interactive chat:

python inference.py

Flag	Default	Description
--prompt	—	One-shot user message
--system	—	Optional system prompt
--temperature	0.7	Sampling temperature
--top-p	0.9	Nucleus sampling
--top-k	50	Top-k
--max-new-tokens	256	Max generation length
--device	cuda if available	cuda / cpu

---

Limitations

• Scale: at ~150M parameters this is a research / edge model, not a frontier system

---

Model family

Model	Stage	Hub
Monostich	SFT (~100M LLaMA)	kerzgrr/Monostich
Monostich-2-base	Pretrain (~150M hybrid)	kerzgrr/Monostich-2-base
Monostich-2	SFT (~150M hybrid)	this repo

---

Citation

@misc{monostich22026,
  title={Monostich-2: A Hybrid GDN-2 + GQA Chat Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Monostich-2}
}

---

Acknowledgments

• flash-linear-attention (Gated DeltaNet-2)
• HuggingFaceTB/smoltalk
• NousResearch/Hermes-3-Dataset
• Base: kerzgrr/Monostich-2-base

---

A monostich is a poem of a single line — small, but complete.
Monostich-2 renews the form.