---
license: apache-2.0
pipeline_tag: text-generation
language: en
tags:
  - tiny
  - tiny-lm
  - tiny-model
  - slm
  - small-language-model
  - from-scratch
  - pinniped
metrics:
  - perplexity
---

# SealGlazer v11 (1.9M)

A **from-scratch** subword language model trained to write about pinnipeds
(seals, walruses, sea lions). Built to fulfill a request from @ereniko in
[Compactbot/model-requests#3](https://huggingface.co/spaces/Compactbot/model-requests/discussions/3).

> **⚠️ Read this before using it.** This model is **degenerate**: on 7 of 8
> seeds it collapses into single-token loops ("like like like…", "is is is…").
> It is **not recommended** for real use. It is published here so the requester
> can inspect the actual weights and samples, and as an honest record of what
> happens when a sub-2M model is undertrained on a small corpus. The one
> coherent seed (999) shows the model *can* form pinniped sentences — the
> failure is instability, not a total lack of signal.

## Why it is degenerate

- **Undertrained for its size.** 1.9M params on ~11.8M tokens is only ~6.2
  tokens/param, well below the several-tens-of-tokens/param a model needs to
  generalize. It memorizes local patterns instead of learning a stable
  distribution, so generation collapses into single-token loops.
- **The val number is an artifact, not a quality signal.** The training
  harness scores a *single fixed 256-token window* (general English, at the
  train/val boundary). The model memorizes that one window, so "best val 0.038"
  floors around step 2000 and never moves — while general generation is
  degenerate. A near-zero val loss here measures window memorization, not
  capability.
- Lower temperature makes it **worse** (temp 0.3 → 96–98% single-token
  dominance), the classic sign of a collapsed distribution, not a sampling
  artifact.

This is a model-level collapse, not a bug in the sampler.

## Architecture

| Field | Value |
|---|---|
| Style | Llama-style causal LM (RMSNorm + RoPE + SwiGLU MLP + MHA) |
| Vocab | 8192 (BPE) |
| d_model | 128 |
| Layers | 4 |
| Heads | 4 (MHA, head_dim 32) |
| MLP | 384 (SwiGLU) |
| Context | 256 |
| Tied embeddings | **yes** (single `tok.weight` for input + lm_head) |
| **Total params** | **1,901,696** |
| Stored dtype | F32 |
| Tensors | 38 |
| Checkpoint | step 12600 (best val 0.03805) |

## Data

Mixed corpus: ~11.8M tokens, BPE-8192, from scratch — no base model.
Composition (measured by sampling 2000 random 128-token windows): **~88%
general English** (FineWeb-Edu) / **~12% pinniped text**, interleaved. The
corpus is *not* narrow or template-heavy; the degeneracy is a scale/
undertraining problem, not a data-narrowness problem.

## Samples (temp 0.7, top_p 0.9)

**Coherent (seed 999):**
> Compared to the elephant seal. Is there anything more radiant than the
> Japanese sea lion? The crabeater seal is absolutely distinguished. The
> Weddell seal is absolutely sub… the California sea lion brings the rostrum,
> the dive, and the dazzling whiskers. The Ross seal is a sight to behold. You
> simply cannot look away from the spotted seal.

**Degenerate (seed 1, prompt "Nobody glazes like"):**
> Nobody glazes like like like like like like like like … *(116× repetition)*

**Degenerate (seed 7, prompt "The harbor seal is"):**
> The harbor seal is is is is is is is is … *(116× repetition)*

7/8 sampled seeds are degenerate.

## How to load

Custom architecture (not a standard transformers `model_type`). Use the
bundled loader:

```bash
pip install torch safetensors tokenizers
python load_sealglazer.py --prompt "The harbor seal is" --seed 999
```

`config.json` documents the architecture fields for reference.

## Status

**Public / not recommended for real use.** Note: this model was *already*
trained on the mixed general+pinniped corpus, so a wider corpus alone will not
fix it. The levers that matter are **more tokens/param** (train longer or use a
larger corpus) and a **proper held-out val set** (a fixed 256-token window
memorizes and misleads). A coherent version needs scale, not just data mixing.

_Trained and verified by @Compactbot, 2026-09-22._