README.md

---
license: apache-2.0
language:
  - en
tags:
  - text-generation
  - causal-lm
  - pytorch
  - pretrain
  - hybrid
  - kimi-delta-attention
  - gated-mla
  - haiku
pipeline_tag: text-generation
library_name: tiny_gdn
datasets:
  - HuggingFaceFW/fineweb-edu
model-index:
  - name: Haiku-base
    results: []
---

<div align="center">

# Haiku-base

### Pretrained base model for the Haiku family (~655M)

[![Model](https://img.shields.io/badge/Model-~655M_params-blue)](.)
[![Stage](https://img.shields.io/badge/Stage-Pretrain_(base)-orange.svg)](.)
[![License](https://img.shields.io/badge/License-Apache_2.0-green.svg)](LICENSE)
[![Architecture](https://img.shields.io/badge/Arch-KDA_+_Gated_MLA-purple.svg)](.)
[![Demo](https://img.shields.io/badge/Space-haiku--demo-indigo.svg)](https://huggingface.co/spaces/kerzgrr/haiku-demo)

*A larger TinyGDN hybrid: Kimi Delta Attention memory plus gated multi-head latent attention*

</div>

---

## What this is

**Haiku-base** is the **pretrained (base) checkpoint** for **Haiku**, the ~655M successor to the Tercet family.

- Scales [`kerzgrr/Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base) from ~502M to ~655M parameters
- Hybrid **Kimi Delta Attention (KDA)** recurrent layers + **gated MLA** (NoPE) full-attention layers
- Own **65,536** BPE tokenizer (not the Tercet 49k vocab)
- This repo is **pretrain-only** raw text continuation
- Chat / instruction SFT is **not released**

This base model is for continuation and research. It will not follow instructions reliably.

---

## Model Architecture

**Pipeline:** `Text Prompt` → `BPE-65K Tokenizer` → `Haiku Hybrid Decoder (36L)` → `Next-token Prediction`

### Hybrid block schedule (×36)

Every 4th layer is gated MLA; the rest are Kimi Delta Attention:

`KDA, KDA, KDA, MLA, …` (3:1 recurrent-to-attention)

| Component | Details |
|-----------|---------|
| **Kimi Delta Attention** | Linear-time recurrent memory (`flash-linear-attention`) |
| **Gated MLA** | DeepSeek-style latent KV, content-only QK (NoPE), full-rank output gate |
| **MLP** | SiTU-GLU |
| **Residuals** | Block attention residual |
| **Norm** | Zero-centered RMSNorm |
| **Embeddings** | Tied input / output |

### Technical specifications

| | |
|--|--|
| **Architecture** | Haiku hybrid (KDA + gated MLA) |
| **Parameters** | 655,270,488 deployable |
| **Hidden size** | 1,024 |
| **Intermediate (MLP)** | 3,840 |
| **Layers** | 36 (27 KDA + 9 gated MLA) |
| **Attention** | 8 heads, Q LoRA rank 512, KV LoRA rank 256 |
| **Linear (KDA)** | 8 heads × 128 dim |
| **Context (trained)** | 2,048 |
| **Max position embeddings** | 32,768 |
| **Vocabulary** | 65,536 (BPE) |
| **RoPE θ** | 1,000,000 (partial factor 0.5; used by KDA) |
| **Precision (Hub weights)** | bfloat16 EMA |
| **Weight file** | `model.safetensors` (~1.22 GiB) |

---

## Training (pretrain)

| | |
|--|--|
| **Dataset** | [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) (10.13B packed train tokens) |
| **Tokens seen** | 4,404,019,200 |
| **Sequence length** | 2,048 |
| **Objective** | Next-token prediction (+ MTP during training; not used at decode) |
| **Optimizer** | Hybrid Muon + AdamW — β₁=0.9, β₂=0.95 |
| **Peak LR** | 2 × 10⁻⁴ |
| **Warmup** | 1% of steps |
| **Grad clip** | 1.0 |
| **EMA** | Karras power EMA (γ=1.0, p=0.75, max decay 0.9999) — **this Hub file is the EMA weights** |
| **Checkpoint** | optimizer step 8,400 |
| **Val loss (EMA)** | 3.6904 (ppl 40.06) |

---

## Install

### 1) System requirements

- Python **3.10+**
- **CUDA GPU strongly recommended**
- PyTorch with CUDA matching your driver

### 2) Create an environment

```bash
python -m venv .venv
# Windows
.venv\Scripts\activate
# Linux / macOS
source .venv/bin/activate
```

### 3) Install PyTorch

Pick the build for your platform from https://pytorch.org. Example:

```bash
pip install torch --index-url https://download.pytorch.org/whl/cu124
```

CPU-only:

```bash
pip install torch
```

### 4) Install Python deps

```bash
pip install safetensors tokenizers huggingface_hub
```

**Flash Linear Attention is installed automatically by `inference.py`** on first run (pinned commit + Windows import patches when needed). Git must be on `PATH`.

### 5) Download the inference script

```bash
curl -L -o inference.py https://huggingface.co/kerzgrr/Haiku-base/resolve/main/inference.py

# or Hugging Face CLI
hf download kerzgrr/Haiku-base inference.py --local-dir .
```

The script auto-downloads `model.safetensors`, `config.json`, `tokenizer.json`, and the `tiny_gdn/` package from this repo.

---

## Quick start

**Single prompt (streams tokens):**

```bash
python inference.py --prompt "The history of computing begins"
```

**Interactive REPL:**

```bash
python inference.py
```

**Common options:**

| Flag | Default | Description |
|------|---------|-------------|
| `--prompt` | *(none)* | One-shot continuation; omit for REPL |
| `--temperature` | `0.8` | Sampling temperature |
| `--top-p` | `0.95` | Nucleus sampling |
| `--top-k` | `50` | Top-k (0 disables) |
| `--max-new-tokens` | `256` | Generation length |
| `--repetition-penalty` | `1.08` | Repetition penalty |
| `--context-length` | `2048` | Tokens kept in the window |
| `--seed` | `42` | RNG seed |
| `--device` | `cuda` if available | `cuda` or `cpu` |
| `--no-stream` | off | Print the full completion at once |
| `--no-bos` | off | Do not prepend `<\|begin_of_text\|>` |
| `--local-dir` | *(none)* | Use a local snapshot directory |

---

## Files

```
kerzgrr/Haiku-base/
  README.md
  inference.py
  requirements.txt
  model.safetensors
  config.json
  tokenizer.json
  tokenizer_config.json
  special_tokens_map.json
  special_token_ids.json
  merges.txt
  vocab.json
  chat_template.jinja
  tiny_gdn/
    __init__.py
    config.py
    model.py
    haiku_layers.py
    nn_common.py
```

---

## Limitations

- **Base model**: not instruction-tuned; may ramble or fail at Q&A format
- **Scale**: ~655M parameters — research / edge prototype, not a frontier model
- **Dependency**: requires `flash-linear-attention` (KDA); not GGUF / llama.cpp compatible today
- **Context**: trained at 2,048; longer windows are experimental

---

## Model family

| Model | Parameters | Architecture | Stage | Hub |
|-------|------------|--------------|-------|-----|
| **Monostich** | ~100M | LLaMA-style | SFT | [`kerzgrr/Monostich`](https://huggingface.co/kerzgrr/Monostich) |
| **Monostich-2-base** | ~150M | TinyGDN hybrid | Pretrain | [`kerzgrr/Monostich-2-base`](https://huggingface.co/kerzgrr/Monostich-2-base) |
| **Monostich-2** | ~150M | TinyGDN hybrid | SFT | [`kerzgrr/Monostich-2`](https://huggingface.co/kerzgrr/Monostich-2) |
| **Couplet-base** | ~268M | TinyGDN hybrid | Pretrain | [`kerzgrr/Couplet-base`](https://huggingface.co/kerzgrr/Couplet-base) |
| **Couplet** | ~268M | TinyGDN hybrid | SFT | [`kerzgrr/Couplet`](https://huggingface.co/kerzgrr/Couplet) |
| **Tercet-base** | ~502M | TinyGDN hybrid | Pretrain | [`kerzgrr/Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base) |
| **Tercet** | ~502M | TinyGDN hybrid | SFT | [`kerzgrr/Tercet`](https://huggingface.co/kerzgrr/Tercet) |
| **Haiku-base** | ~655M | KDA + gated MLA | Pretrain | *this repo* |

---

## Citation

```bibtex
@misc{haikubase2026,
  title={Haiku-base: A 655M Hybrid KDA + Gated-MLA Language Model},
  author={kerzgrr},
  year={2026},
  url={https://huggingface.co/kerzgrr/Haiku-base}
}
```

---

## Acknowledgments

- [flash-linear-attention](https://github.com/fla-org/flash-linear-attention) (Kimi Delta Attention)
- [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu)
- Tercet family: [`kerzgrr/Tercet-base`](https://huggingface.co/kerzgrr/Tercet-base)
- PyTorch SDPA / Hugging Face Hub + tokenizers

---

<div align="center">

*A haiku is three lines — larger than a tercet, still compact.*

</div>