---
license: apache-2.0
language:
- en
datasets:
- allenai/c4
tags:
- pytorch
- transformers
- gpt
- yatnmn
- nmn
- chinchilla
- nanochat
- text-generation
- ablation
library_name: transformers
pipeline_tag: text-generation
---

# YatNMN-Softplus + constant α=1 d=12 Chinchilla (261M) — PyTorch

A 261M-parameter **nanochat-architecture GPT** with YatNMN-Softplus MLP where **α is fixed at 1** (non-learnable). This is the **α ablation** — testing whether the learnable per-layer scaling in YatNMN is load-bearing.

**Result: learnable α is critical.** Fixing α=1 costs **+0.12 nats** on C4 and the model generalizes much worse on wikitext (PPL 67 vs 40).

## Ablation table (d=12, 261M, Chinchilla 20×, 3-seed mean)

| Variant | C4 smooth | wikitext PPL | vs GELU |
|---|---|---|---|
| YatNMN per-neuron + learnable α | **2.98** | **40.15** | **−0.14** |
| YatNMN scalar_bias + learnable α | 3.06 | 39.53 | −0.06 |
| **YatNMN per-neuron + constant α=1 (this)** | **3.10** | **67.09** | **−0.02** |
| YatNMN sb + constant α=1 | 3.09 | 78.34 | −0.03 |
| GELU | 3.12 | 46.52 | baseline |

## Quick start

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "mlnomad/yatnmn-softplus-ca-d12-chinchilla-261M-pytorch",
    trust_remote_code=True, dtype=torch.float32,
).eval()
tokenizer = AutoTokenizer.from_pretrained("mistralai/Mistral-7B-v0.1")

ids = tokenizer("The meaning of life is", return_tensors="pt").input_ids
with torch.no_grad():
    out = model.generate(ids, max_new_tokens=50, do_sample=False,
                         use_cache=True, pad_token_id=tokenizer.eos_token_id or 0)
print(tokenizer.decode(out[0], skip_special_tokens=True))
```

## Config

Per-neuron `(ff,)` bias + softplus_bias + learnable_epsilon + **constant_alpha=True (α=1 fixed)**.

| | |
|---|---|
| **Parameters** | 261,133,214 |
| **Final smooth loss** | 3.08 (3-seed mean 3.10 ± 0.02) |
| **Wikitext-103 PPL** | 67.09 |
| **Training data** | `allenai/c4`, 5.22 B tokens (Chinchilla 20×) |
| **Hardware** | TPU v6e-8, europe-west4-a |

## Related

- [`mlnomad/yatnmn-softplus-d12-chinchilla-261M-pytorch`](https://huggingface.co/mlnomad/yatnmn-softplus-d12-chinchilla-261M-pytorch) — with learnable α (loss 2.98, PPL 40)
- [`mlnomad/gelu-d12-chinchilla-261M-pytorch`](https://huggingface.co/mlnomad/gelu-d12-chinchilla-261M-pytorch) — GELU baseline

## License

Apache 2.0.