README.md

---
license: apache-2.0
language:
  - en
tags:
  - text-generation
  - causal-lm
  - pytorch
  - sft
  - instruction-tuned
  - chat
  - hybrid
  - gated-deltanet
  - gqa
  - monostich
pipeline_tag: text-generation
library_name: tiny_gdn
datasets:
  - HuggingFaceTB/smoltalk
  - NousResearch/Hermes-3-Dataset
base_model: kerzgrr/Monostich-2-base
model-index:
  - name: Monostich-2
    results: []
---

<div align="center">

# Monostich-2

### Instruction-tuned chat model (~150M) — Monostich-2 family

[![Model](https://img.shields.io/badge/Model-~150M_params-blue)](.)
[![Stage](https://img.shields.io/badge/Stage-SFT_(chat)-green.svg)](.)
[![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](LICENSE)
[![Base](https://img.shields.io/badge/Base-Monostich--2--base-orange.svg)](https://huggingface.co/kerzgrr/Monostich-2-base)

*Second-generation Monostich-class model — hybrid GDN-2 + GQA, chat-tuned*

</div>

---

## What this is

**Monostich-2** is the **supervised fine-tuned (SFT) chat checkpoint** for the Monostich-2 family.

- Base (pretrain): [`kerzgrr/Monostich-2-base`](https://huggingface.co/kerzgrr/Monostich-2-base)
- Successor to [`kerzgrr/Monostich`](https://huggingface.co/kerzgrr/Monostich) (~100M LLaMA-style)
- Architecture: hybrid **Gated DeltaNet-2** + **gated GQA**

---

## Training

### Pretrain → SFT

| Stage | Details |
|-------|---------|
| **Base** | FineWeb-Edu pretrain → [`Monostich-2-base`](https://huggingface.co/kerzgrr/Monostich-2-base) |
| **SFT mix** | [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk) + [NousResearch/Hermes-3-Dataset](https://huggingface.co/datasets/NousResearch/Hermes-3-Dataset) |
| **Epochs** | 1 full epoch |
| **Wall time** | **21.70 hours** |
| **Final step** | optimizer step **10619** |
| **Weights** | EMA (Hub `model.safetensors` is EMA @ bfloat16) |
| **Seq length** | 8192 (packed SFT) |
| **Peak LR** | 1 × 10⁻⁴ AdamW, cosine → 10% min |
| **Final val loss (EMA)** | **1.532** (ppl ≈ 4.63) |

### Chat template (ChatML)

```
<|begin_of_text|><|im_start|>system
{system}<|im_end|>
<|im_start|>user
{user}<|im_end|>
<|im_start|>assistant
{assistant}<|im_end|>
```

Generation prompt ends at `<|im_start|>assistant\n`.

---

## Model Architecture

Same TinyGDN hybrid as the base (~149.1M params):

| | |
|--|--|
| **Layers** | 32 (GDN-2 ×3 + GQA every 4th) |
| **Hidden** | 512 |
| **MLP** | SwiGLU 1472 |
| **Attention** | 4 Q / 1 KV, head dim 128, partial RoPE |
| **Linear** | Gated DeltaNet-2, 4 heads × 128 |
| **Vocab** | 49,152 BPE |

---

## Install & run

```bash
pip install torch safetensors tokenizers huggingface_hub
hf download kerzgrr/Monostich-2 inference.py --local-dir .
python inference.py --prompt "What is the capital of France?"
```

`inference.py` auto-downloads weights/tokenizer/`tiny_gdn/` and **auto-installs** pinned `flash-linear-attention` (Windows applies Hub patches). Git required on `PATH`.

**Interactive chat:**

```bash
python inference.py
```

| Flag | Default | Description |
|------|---------|-------------|
| `--prompt` | — | One-shot user message |
| `--system` | — | Optional system prompt |
| `--temperature` | `0.7` | Sampling temperature |
| `--top-p` | `0.9` | Nucleus sampling |
| `--top-k` | `50` | Top-k |
| `--max-new-tokens` | `256` | Max generation length |
| `--device` | `cuda` if available | `cuda` / `cpu` |

---

## Limitations

- **Scale**: at ~150M parameters this is a research / edge model, not a frontier system

---

## Model family

| Model | Stage | Hub |
|-------|-------|-----|
| Monostich | SFT (~100M LLaMA) | [`kerzgrr/Monostich`](https://huggingface.co/kerzgrr/Monostich) |
| Monostich-2-base | Pretrain (~150M hybrid) | [`kerzgrr/Monostich-2-base`](https://huggingface.co/kerzgrr/Monostich-2-base) |
| **Monostich-2** | **SFT (~150M hybrid)** | **this repo** |

---

## Citation

```bibtex
@misc{monostich22026,
  title={Monostich-2: A Hybrid GDN-2 + GQA Chat Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Monostich-2}
}
```

---

## Acknowledgments

- [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) (Gated DeltaNet-2)
- [HuggingFaceTB/smoltalk](https://huggingface.co/datasets/HuggingFaceTB/smoltalk)
- [NousResearch/Hermes-3-Dataset](https://huggingface.co/datasets/NousResearch/Hermes-3-Dataset)
- Base: [`kerzgrr/Monostich-2-base`](https://huggingface.co/kerzgrr/Monostich-2-base)

---

<div align="center">

*A monostich is a poem of a single line — small, but complete.*  
*Monostich-2 renews the form.*

</div>