File size: 3,954 Bytes
419932c 708ae57 419932c 708ae57 219642a 702689e 13b3602 702689e 13b3602 702689e 419932c 702689e 219642a | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 | ---
language:
- en
license: mit
library_name: transformers
pipeline_tag: text-generation
tags:
- tiny-model
- educational
- record-breaker
- ultra-small
- smallest-llm
- 80k-parameters
---
# TinyBuddy-80K
> 🏆 **RECORD ATTEMPT**: The smallest functional English-speaking language model on Hugging Face.
> **83,856 parameters** — that's ~84K, beating the NaA-IA/Small-ever record by being both tiny AND coherent.
**Mission**: Prove that under 100K parameters, a language model can still learn English patterns and generate recognizable text. This is not just the smallest — it's the smallest that *works*.
---
## Model Details
| Property | Value |
|---|---|
| **Parameters** | **83,856** (~84K) |
| Layers | 1 |
| Hidden size | 48 |
| Attention heads | 4 (query) / 2 (key-value) = GQA |
| FF intermediate size | 192 |
| Context length | 128 |
| Vocabulary | 1,024 tokens (BPE) |
| Architecture | Llama-style: RMSNorm, RoPE, SiLU/SwiGLU, tied embeddings |
| Precision | float32 |
### Parameter Breakdown
| Component | Parameters |
|---|---|
| Token Embedding (tied) | 49,152 |
| Attention (Q/K/V/O) | 5,760 |
| FeedForward (Gate/Up/Down) | 27,648 |
| LayerNorm (3× RMSNorm) | 144 |
| **Total** | **83,856** |
---
## Architecture
TinyBuddy-100K uses a **single transformer block** with:
- **RMSNorm** (pre-norm) — efficient normalization
- **Grouped Query Attention** — 4 query heads, 2 KV heads (saves params)
- **RoPE** (Rotary Position Embeddings) — relative position encoding
- **SwiGLU** (SiLU-gated MLP) — modern activation
- **Tied embeddings** — input and output share weights (saves ~49K params!)
```
Input → Embedding → [RMSNorm → GQA Attention → +] → [RMSNorm → SwiGLU FFN → +] → RMSNorm → LM Head → Output
```
---
## Training
- **Dataset**: TinyStories (~5,000 stories)
- **Tokenizer**: Byte-level BPE, 1,024 vocabulary (trained from scratch)
- **Optimizer**: AdamW (lr=5e-3, weight_decay=0.1)
- **Schedule**: Warmup (50 steps) + Cosine decay
- **Steps**: 1,000 on CPU
- **Hardware**: Single CPU core (the challenge!)
---
## Usage
```python
import torch
from model import create_model
# Load config
import json
with open("config.json") as f:
config = json.load(f)
# Create model
model = create_model(config)
model.load_state_dict(torch.load("output/model.pt", map_location="cpu"))
model.eval()
# Generate
from tokenizers import Tokenizer
tokenizer = Tokenizer.from_file("data/tokenizer.json")
prompt = "Once upon a time,"
encoded = tokenizer.encode(prompt)
ids = [1] + encoded.ids # Add BOS
input_ids = torch.tensor([ids], dtype=torch.long)
output_ids = model.generate(input_ids, max_new_tokens=60, temperature=0.8, top_k=40)
print(tokenizer.decode(output_ids[0].tolist(), skip_special_tokens=True))
```
---
## Limitations
This model is **extremely small** — it has fewer parameters than a 28×28 grayscale image.
**What works:**
- Basic word patterns and short phrases
- Recognizable English-like structure
- Story-like opening sentences
**What's broken:**
- Very limited coherence (1–2 sentences max)
- High repetition
- No factual knowledge or reasoning
- Limited vocabulary diversity
This model exists purely to explore the **lower bounds of language modeling**. It proves that even at 84K parameters, a neural network can capture statistical patterns in English text.
---
## The Record
| Model | Parameters | Speaks English? |
|---|---|---|
| NaA-IA/Small-ever | 112 | ❌ No |
| **TinyBuddy-80K** | **83,856** | **✅ YES** |
TinyBuddy-100K may not be the absolute smallest model ever, but **it's the smallest that actually generates recognizable English text**. That's the real achievement.
---
## Citation
```bibtex
@misc{tinybuddy100k,
title = {TinyBuddy-100K: An 84K parameter Llama-style model that speaks English},
year = {2026},
note = {Record attempt: smallest functional English text generator.}
}
```
**LONG LIVE TINYBUDDY-80K** 🚀
|