README.md

---
license: cc-by-4.0
language:
- en
pipeline_tag: text-generation
library_name: pytorch
tags:
- pytorch
- text-generation
- causal-lm
- recurrent-transformer
- xsa
- grouped-query-attention
- small-language-models
- from-scratch
datasets:
- HuggingFaceFW/finephrase
- HuggingFaceTB/finemath
model-index:
- name: Nero XS 1.1
  results:
  - task:
      type: text-generation
      name: Zero-shot multiple-choice language modeling
    dataset:
      name: HellaSwag
      type: hellaswag
    metrics:
    - type: accuracy_norm
      value: 0.2817167894841665
      verified: false
  - task:
      type: text-generation
      name: Zero-shot multiple-choice language modeling
    dataset:
      name: ARC-Easy
      type: ai2_arc
      config: ARC-Easy
    metrics:
    - type: accuracy_norm
      value: 0.3063973063973064
      verified: false
  - task:
      type: text-generation
      name: ARC-Challenge
    dataset:
      name: ARC-Challenge
      type: ai2_arc
      config: ARC-Challenge
    metrics:
    - type: accuracy_norm
      value: 0.22098976109215018
      verified: false
  - task:
      type: text-generation
      name: PIQA
    dataset:
      name: PIQA
      type: piqa
    metrics:
    - type: accuracy_norm
      value: 0.529923830250272
      verified: false
  - task:
      type: text-generation
      name: ArithMark-3
    dataset:
      name: ArithMark-3
      type: AxiomicLabs/Arithmark-3.0
    metrics:
    - type: accuracy_norm
      value: 0.324
      verified: false
---

# Nero XS 2

A 2,993,152-parameter recurrent grouped-query XSA language model trained from scratch with PyTorch/XLA on TPU v5e-8.

## Quick start (PyTorch)

```bash
pip install "torch>=2.5" "transformers>=5" safetensors huggingface_hub
```

```python
from huggingface_hub import snapshot_download
import sys

model_dir = snapshot_download("j0no12/Nero-XS-2")
sys.path.insert(0, model_dir)

from modeling_nero_xs_2 import load_model, generate

model, tokenizer = load_model(model_dir, device="cpu")
text = generate(
    model,
    tokenizer,
    prompt="The future of small language models is",
    max_new_tokens=64,
    temperature=0.8,
    top_p=0.95,
    repetition_penalty=1.1,
    seed=7,
)
print(text)
```

Nero XS 2 uses the custom PyTorch class `NeroXSAForCausalLM`. It is not a drop-in Transformers `AutoModelForCausalLM` checkpoint. The complete standalone implementation is included in `modeling_nero_xs_2.py`.

## Results

All scores are zero-shot normalized accuracy on the full evaluation split.

| Benchmark | Examples | Nero XS 2 | Nero XS |
|---|---:|---:|---:|
| HellaSwag | 10,042 | **28.17%** | 27.38% |
| ARC-Easy | 2,376 | 30.64% | **30.98%** |
| ARC-Challenge | 1,172 | **22.10%** | 20.73% |
| PIQA | 1,838 | 52.99% | **53.86%** |
| ArithMark-3 | 1,000 | **32.40%** | 32.10% |
| **Unweighted mean** | — | **33.26%** | 33.01% |

HellaSwag, ARC, and PIQA were evaluated with `lm-eval` 0.4.12 using exact continuation likelihood, zero few-shot examples, `acc_norm`, and full task splits. ArithMark-3 used the official independent-tokenization continuation protocol and `acc_norm`. Raw results are included under `evals/`.

**Takeaway:** this recovery checkpoint improves four of five public tasks over the original Nero-XS-2 checkpoint and raises the unweighted mean 0.25 percentage points above Nero XS. It is not established as a top-three or state-of-the-art model. Frozen-dev NLL worsened during recovery even as public multiple-choice scores improved, so these gains should be interpreted cautiously.

## Architecture

| Component | Configuration |
|---|---|
| Architecture class | `NeroXSAForCausalLM` |
| Stored parameters | **2,993,152** |
| Physical / effective blocks | 10 / 14 |
| Recurrent layout | 1 prelude + 4 middle blocks × 2 passes + 5 coda |
| Residual width | 128 |
| Attention | 4 query heads, 2 key/value heads, head dimension 32 |
| Attention modification | XSA value-subtraction after causal attention |
| Attention normalization | Per-head Q/K RMSNorm with RoPE |
| Recurrent conditioning | Learned pass embeddings and channel-wise refresh gates |
| MLP | Gated dense MLP, width 582 |
| Vocabulary | 2,048-token byte-level BPE |
| Embeddings | Tied input/output table |
| Position signal | RoPE, base 20,000 |

The model shares four physical middle blocks across two recurrent passes. Each pass has a learned embedding and channel-wise gate. XSA subtracts the component of each attended head output aligned with the current-token value vector.

## Training story

### 1. Nero-XS-2 pretraining

- Exact target: **25,000,148,992 tokens**.
- Sequence length 256; global batch 2,048 sequences, or 524,288 tokens per optimizer step.
- AdamW state, scheduler position, sampler position, and RNG state were preserved across TPU sessions.
- The original 2,994,179-parameter graph included EngramLite hashed bigram/trigram memory and FFN width 531.
- Original frozen-dev NLL: **2.4375**.

### 2. Engram-anneal recovery

- Preserved 84 compatible tensors from the 25B checkpoint.
- Expanded 30 FFN tensors from width 531 to 582, copying every trained channel and randomly initializing only 51 new channels.
- Froze and annealed EngramLite to zero over **1,500,037,632 tokens**.
- Continued the final Engram-free graph through **2,500,329,472 recovery tokens**.
- Total recorded exposure: **27,500,478,464 tokens**.
- Recovery throughput: 493,879 tokens/s on TPU v5e-8.
- Final frozen-dev NLL: **2.65625**, worse than before recovery.

The released checkpoint is the final recovery checkpoint. Public tasks were evaluated only after the recovery design and checkpoint were fixed.

## Data lineage

The immutable prepared corpus contained 495,134,047 unique training tokens and was sampled repeatedly to the training budget.

| Source | Intended mixture | Pinned revision |
|---|---:|---|
| FinePhrase | 45% | `78cf4a5ed0099214979c094c963e699c19163838` |
| FineMath 4+ | 30% | `e92b25a616738fe95dc186b64dfb19f9c8525594` |
| CoRNStack positive Python | 15% | `25fb04bd3537983a622d01104a967a5a7f9eaef8` |
| DCLM replay | 10% | locally pinned source stream |

Preparation rejected any document sharing a normalized 13-word sequence with HellaSwag, ARC, PIQA, or ArithMark-3 prompts or answers. The prepared training stream SHA-256 is `cd3ac872bf365c364f82348703635613fb78b4aa927b7e9f0d7db1a89c729ebf`.

## Evaluation details

| Item | Value |
|---|---|
| Released checkpoint | Recovery checkpoint at 27,500,478,464 tokens |
| `lm-eval` version | 0.4.12 |
| Few-shot examples | 0 |
| Multiple-choice metric | `acc_norm` |
| Evaluation maximum context | 2,048 tokens (`lm-eval`); 1,024 for ArithMark-3 |
| ArithMark-3 SHA-256 | `bf8ab1a5193d52cdf0e05ff0b3ca226bdfcf416cb6e75562dcbe72e7e4559435` |

## Reproducibility

| Item | Value |
|---|---|
| Framework | PyTorch/XLA |
| Hardware | TPU v5e-8 |
| Preparation seed | 20260904 |
| Tokenizer SHA-256 | `5da5a6a51f165486e3e3ed9371bab510374d3d478ff105528e87fd8cfecd2705` |
| Portable source checkpoint SHA-256 | `90b575942b402f088f76c3e1a717230b5d47b54635f43947910748bf82e08ed0` |
| Released safetensors SHA-256 | `3dd9650bf613dffc1ae1c1c2f934ad9343a1e96aaacb8243dbbbee4b1d99e7df` |

The released safetensors file was produced directly from the portable final checkpoint. The standalone implementation was checked against the training implementation on a fixed input before release.

## Limitations

- Fewer than three million parameters sharply limits factual recall, instruction following, coherence, and long-form generation.
- Training used 256-token sequences. Longer contexts are outside the established training distribution.
- The 2K tokenizer favors parameter efficiency over tokenization efficiency.
- Generation uses full-prefix recomputation and has no KV cache.
- This is a base model, not an instruction-tuned chat assistant.
- Recovery worsened frozen-dev NLL from 2.4375 to 2.65625 despite improving four public tasks.
- No independent human evaluation or comprehensive safety evaluation is published.
- The scores are not independently verified leaderboard submissions.

## Files

| File | Description |
|---|---|
| `model.safetensors` | FP32 model weights |
| `modeling_nero_xs_2.py` | Standalone architecture, loading, and generation code |
| `config.json` | Exact released architecture configuration |
| `tokenizer.json`, `tokenizer_config.json` | 2K byte-level BPE tokenizer |
| `training_state.json` | Training and recovery summary |
| `evals/` | Raw benchmark outputs |

## Citation

```bibtex
@misc{neroxs2_2026,
  author       = {j0no12},
  title        = {Nero XS 2: A 2.993M-Parameter Recurrent Grouped-Query XSA Language Model},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/j0no12/Nero-XS-2}},
  note         = {Trained from scratch with PyTorch/XLA on TPU v5e-8}
}
```

## License

The original Nero XS 2 weights and repository material are released under CC-BY-4.0. Attribution is required. Upstream dataset terms and attribution requirements remain applicable to their respective source material.