README.md

---
license: mit
language:
  - en
library_name: pytorch
tags:
  - gpt
  - from-scratch
  - nanogpt
  - language-model
  - educational
datasets:
  - Skylion007/openwebtext
pipeline_tag: text-generation
---

# slm-from-scratch-77m

A 77M parameter GPT-style language model written and trained from scratch in PyTorch on a single RTX 5070 Ti. No HuggingFace `transformers` model classes were used. Every component (attention, MLP, blocks, embeddings, the training loop) was hand-coded as part of a learning project to understand the full LLM training pipeline.

This is Phase 1 of a larger build documented at [github.com/Ishaanred/slm-from-scratch](https://github.com/Ishaanred/slm-from-scratch). It is a deliberately small, deliberately undertrained checkpoint. It is a portfolio and learning artifact, not a production model.

## What it is

| | |
|---|---|
| Parameters | 77.2M |
| Architecture | Decoder-only GPT (pre-LayerNorm) |
| Layers | 8 |
| Heads | 8 |
| Embedding dim | 512 |
| Context length | 1024 |
| Vocabulary | GPT-2 BPE (50,257) |
| Attention | Flash Attention via PyTorch SDPA |
| Precision | weights stored fp32, trained in bf16 |

## Training

| | |
|---|---|
| Data | OpenWebText |
| Steps | 5,000 |
| Tokens seen | ~80M |
| Optimizer | AdamW, cosine LR with warmup, weight decay on 2D params |
| Hardware | 1x RTX 5070 Ti (16GB) |
| Wall time | ~20 min |
| Validation loss | 5.24 |

## Honest assessment

This model is undertrained on purpose. At ~80M tokens it has seen roughly 5% of what a 77M model needs to converge (Chinchilla suggests ~1.5B tokens for this size). The output is grammatical English with plausible local structure but no coherence across sentences.

A companion 50M model trained on the same 80M tokens reached a lower validation loss (5.12). At a fixed, small token budget the smaller model wins, because the larger one is more undertrained relative to its capacity. That gap is expected to reverse with longer training. Phase 3 of the project runs the controlled scaling-law experiments that test this. See [slm-from-scratch-phase3-scaling](https://huggingface.co/redredredredredred/slm-from-scratch-phase3-scaling) for those checkpoints and the scaling-law results.

### Sample output

Prompt: `The meaning of life is`

```
The meaning of life is the first ever-to-to-mused of self-lab and the first time period of time.
The difference between these systems, and even the potential difference to be an example of these
two seasons. The reason it might be, but the first time that it's very well worth noting that we
can build up a new one as a result of the whole way we're talking about...
```

## Files

- `model.safetensors` — model weights, fp32, optimizer state stripped
- `model.py` — the from-scratch model definition needed to load these weights
- `config.json` — architecture and training metadata

## Usage

This is not a `transformers` architecture, so load it with the included `model.py`.

```python
import json, torch
from safetensors.torch import load_file
from model import GPT, GPTConfig

cfg = json.load(open("config.json"))
model = GPT(GPTConfig(
    n_layer=cfg["n_layer"], n_head=cfg["n_head"], n_embd=cfg["n_embd"],
    block_size=cfg["block_size"], vocab_size=cfg["vocab_size"], dropout=0.0,
))
model.load_state_dict(load_file("model.safetensors"))
model.eval()

# tokenize with GPT-2 BPE, e.g. via tiktoken:
import tiktoken
enc = tiktoken.get_encoding("gpt2")
ids = torch.tensor([enc.encode("The meaning of life is")])

with torch.no_grad():
    for _ in range(50):
        logits, _ = model(ids[:, -cfg["block_size"]:])
        nxt = torch.softmax(logits[:, -1, :] / 0.8, dim=-1).multinomial(1)
        ids = torch.cat([ids, nxt], dim=1)
print(enc.decode(ids[0].tolist()))
```

## Limitations

- Tiny and undertrained. Not suitable for any real task.
- Trained on OpenWebText, so it inherits the biases and noise of unfiltered web text scraped from Reddit-linked URLs.
- No instruction tuning, no safety alignment. It is a raw next-token predictor.

## License

MIT. Built as an educational project.