mlnomad/yatnmn-softplus-ca-d12-chinchilla-261M-pytorch

YatNMN-Softplus + constant α=1 d=12 Chinchilla (261M) — PyTorch

A 261M-parameter nanochat-architecture GPT with YatNMN-Softplus MLP where α is fixed at 1 (non-learnable). This is the α ablation — testing whether the learnable per-layer scaling in YatNMN is load-bearing.

Result: learnable α is critical. Fixing α=1 costs +0.12 nats on C4 and the model generalizes much worse on wikitext (PPL 67 vs 40).

Ablation table (d=12, 261M, Chinchilla 20×, 3-seed mean)

Variant	C4 smooth	wikitext PPL	vs GELU
YatNMN per-neuron + learnable α	2.98	40.15	−0.14
YatNMN scalar_bias + learnable α	3.06	39.53	−0.06
YatNMN per-neuron + constant α=1 (this)	3.10	67.09	−0.02
YatNMN sb + constant α=1	3.09	78.34	−0.03
GELU	3.12	46.52	baseline

Quick start

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "mlnomad/yatnmn-softplus-ca-d12-chinchilla-261M-pytorch",
    trust_remote_code=True, dtype=torch.float32,
).eval()
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

ids = tokenizer("The meaning of life is", return_tensors="pt").input_ids
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=50, do_sample=False,
                         use_cache=True, pad_token_id=tokenizer.eos_token_id or 0)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Config

Per-neuron (ff,) bias + softplus_bias + learnable_epsilon + constant_alpha=True (α=1 fixed).

Parameters	261,133,214
Final smooth loss	3.08 (3-seed mean 3.10 ± 0.02)
Wikitext-103 PPL	67.09
Training data	allenai/c4, 5.22 B tokens (Chinchilla 20×)
Hardware	TPU v6e-8, europe-west4-a

Related

• mlnomad/yatnmn-softplus-d12-chinchilla-261M-pytorch — with learnable α (loss 2.98, PPL 40)
• mlnomad/gelu-d12-chinchilla-261M-pytorch — GELU baseline

License

Apache 2.0.