---
license: apache-2.0
language:
- en
library_name: pytorch
pipeline_tag: text-generation
tags:
- custom-architecture
- mixture-of-experts
- kuramoto
- wheeler-dewitt
- grpo
- spikewhale
- research
- experimental
---
# Chimera-64M
> **Research artifact.** Experiment, not a product. Don't use it for anything real unless it is trained much longer (see *Intended use & limitations*).
~64M from-scratch decoder. Capstone of the family that asks whether a transformer can generalize if the FFN is a weird dynamical system. Chimera doesn't pick one: per-layer mixer is a **council of three physics cores** - coupled oscillators, fungal-colony growth, and the Wheeler-DeWitt wave equation. A small **GRPO** rule learns *which law of physics to apply to each token*.
## Model summary
| | |
|---|---|
| **Parameters** | ~64.0M |
| **Type** | Decoder-only autoregressive LM (custom `QuazimotoLM`) |
| **Layers / hidden** | 10 / 768 |
| **Attention** | MLA + partial RoPE + QK-Norm + GQA + **Elo/Bradley-Terry ratings** |
| **Channel mixer** | **ChimeraBlock** - GRPO council over 3 physics cores |
| **Context** | `block_size` 1024 (RoPE `max_position_embeddings` 4096) |
| **Vocab** | 16,512 (custom SpikeWhale byte-merge tokenizer) |
| **Traits** | HRM refinement, MoE-SwiGLU, MTP, JEPA (train-time), fractal phase seed |
Two checkpoints are included:
- `chkpt/quazimoto.pt` - **base**, pretraining step **40,000** (continued at block 1024).
- `chkpt/quazimoto_sft.pt` - **chat SFT**, step **4,250**.
## The ChimeraBlock - a GRPO council of physics
Instead of an MLP, each layer runs **three physics cores in parallel**, each proposing a candidate update of the hidden state, then combines them with a differentiable, in-forward-pass GRPO step:
- **KuramotoCore** - a bank of coupled phase oscillators (Kuramoto dynamics), integrated by a few differentiable Euler steps. *(from Quazimoto/Positronic)*
- **GrowthCore** - a Neighbour-Sensing fungal-colony growth model: tips grow in a bounded latent region, sensing their own density and steering away from it. *(from Mycel)*
- **WaveCore** - the **Wheeler-DeWitt** equation: `wdw_modes` minisuperspace modes under a learnable **Lorentzian** supermetric, run as a leapfrog wave; exposes a Hamiltonian-constraint aux loss `⟨H²⟩`. *(from Wheeler)*
**The GRPO council** (the meta-mixer, from grpo_lm) then decides the blend, per token:
1. an internal **critic** scores each candidate → reward `r_g`;
2. **group-relative advantage** `A_g = (r_g - mean)/(std + ε)`, clamped;
3. selection weights `softmax(A/temp)`, **clipped around uniform** (the PPO clip);
4. **anchored** back toward uniform (the KL-to-reference anchor);
5. output = the resulting convex combination of the three candidates.
So the network **learns which mechanism to trust for each token** rather than committing to one. Everything else is the shared "SpikeWhale/Byrne family": MLA attention with Elo/Bradley-Terry key ratings, HRM refinement, MoE-SwiGLU, MTP heads, a JEPA objective, and the SpikeWhale tokenizer. Each trait is a gated near-no-op at init that activates only if it helps.
## Usage
Self-contained - load with the bundled code:
```python
import torch
from model import QuazimotoLM, QuazimotoConfig
from spike_tokenizer import SpikeTokenizer
tok = SpikeTokenizer(vocab_file="tokenizer.json")
ck = torch.load("chkpt/quazimoto.pt", map_location="cpu", weights_only=False)
cfg = QuazimotoConfig(**ck["family_config"])
model = QuazimotoLM(cfg); model.load_state_dict(ck["model"], strict=False); model.eval()
ids = torch.tensor([tok.encode("The history of science", add_special_tokens=False)])
out = model.generate(ids, 80, temperature=0.8, top_k=40, use_cache=True)
print(tok.decode(out[0].tolist(), skip_special_tokens=True))
```
Also bundled: `generate.py` (KV cache + self-speculative decoding), `chat_sft.py` (ChatML REPL), and `visualize.py` (a live GRPO-council dashboard showing which physics law wins each token).
## Intended use & limitations
**This is a research artifact for studying architecture generalization - not a usable assistant.**
- The **base** produces fluent, locally-coherent free-form text, but is not factual or reliable over long spans.
- The **SFT** checkpoint (only 4,250 steps) stays on-topic and produces structured, math-formatted answers, but **hallucinates** (its formulas and facts are unreliable) and does not truly follow instructions. This is a capacity-and-scale limit of a 64M model with a short SFT run.
- **Do not deploy this.** To be actually useful it needs to be trained substantially longer (more tokens, larger, longer SFT). It is shared to demonstrate that three exotic physics mixers, GRPO-selected per token, *can* learn language - the point is *that it generalizes*, not *what it knows*.
- Trained on public web/edu/math corpora; inherits their biases and can produce incorrect or offensive text.
## Family & thesis
Chimera is the synthesis of a controlled series of mixer swaps (Kuramoto → Quazimoto/Positronic; Neighbour-Sensing growth → Mycel; Wheeler-DeWitt → Wheeler). The shared thesis: *the specific mixing mechanism matters far less than the loop of attend → compress → repeat* - and a network can even **learn to arbitrate between several** exotic mechanisms per token.
## Citation
```bibtex
@misc{byrne2026chimera,
title = {Chimera-64M: a language model whose channel mixer is a GRPO council of
coupled oscillators, fungal-colony growth, and the Wheeler--DeWitt equation},
author = {Byrne, Dean},
year = {2026},
note = {Research artifact; custom PyTorch architecture (Quazim0t0 / SpikeWhale family)}
}
```
## Update: format-blended SFT
Short instruction/format SFT on a 60/25/15 blend of HuggingFaceTB/smoltalk,
GSM8K-train ('#### N' reasoning), and MMLU-style ('Answer: ') examples,
so chat fluency improves while the benchmark output-formats are preserved.
Held-out (test-split) before->after:
held-out 2.23->1.34, MMLU acc 0.035->0.250, format 0.140->1.000
Note: fluency + output-format gains. Benchmark *accuracy* remains near the floor
for a model this size -- the SFT does not add reasoning ability.