README.md

---
license: cc-by-4.0
language:
- en
pipeline_tag: text-generation
library_name: mlx
tags:
- mlx
- text-generation
- causal-lm
- recurrent-transformer
- xsa
- small-language-models
- from-scratch
datasets:
- HuggingFaceFW/finephrase
model-index:
- name: Nero XS
  results:
  - task:
      type: text-generation
      name: Zero-shot multiple-choice language modeling
    dataset:
      name: HellaSwag
      type: hellaswag
    metrics:
    - type: accuracy_norm
      value: 0.27384983071101376
      verified: false
  - task:
      type: text-generation
      name: Zero-shot multiple-choice language modeling
    dataset:
      name: ARC-Easy
      type: ai2_arc
      config: ARC-Easy
    metrics:
    - type: accuracy_norm
      value: 0.30976430976430974
      verified: false
  - task:
      type: text-generation
      name: Zero-shot multiple-choice language modeling
    dataset:
      name: ARC-Challenge
      type: ai2_arc
      config: ARC-Challenge
    metrics:
    - type: accuracy_norm
      value: 0.20733788395904437
      verified: false
  - task:
      type: text-generation
      name: Zero-shot multiple-choice language modeling
    dataset:
      name: PIQA
      type: piqa
    metrics:
    - type: accuracy_norm
      value: 0.5386289445048966
      verified: false
  - task:
      type: text-generation
      name: Zero-shot multiple-choice language modeling
    dataset:
      name: ArithMark-3
      type: AxiomicLabs/Arithmark-3.0
    metrics:
    - type: accuracy_norm
      value: 0.321
      verified: false
---

# Nero XS

A 2,996,480-parameter recurrent XSA language model trained from scratch with MLX on Apple Silicon.

## Quick start (MLX)

```bash
pip install "mlx>=0.32" "transformers>=5" huggingface_hub numpy
```

```python
from huggingface_hub import snapshot_download
import sys

model_dir = snapshot_download("j0no12/Nero-XS")
sys.path.insert(0, model_dir)

from modeling_nero_xs_mlx import load_model, generate

model, tokenizer = load_model(model_dir)
text = generate(
    model,
    tokenizer,
    prompt="The future of small language models is",
    max_new_tokens=64,
    temperature=0.8,
    top_p=0.95,
    repetition_penalty=1.1,
    seed=7,
)
print(text)
```

Nero XS uses the custom MLX architecture class `NeroXSAForCausalLM`. It is not a drop-in Transformers `AutoModelForCausalLM` checkpoint. The complete standalone inference implementation is included in `modeling_nero_xs_mlx.py`.

## Results

All scores are zero-shot normalized accuracy on the full evaluation split.

| Benchmark | Examples | Nero XS |
|---|---:|---:|
| HellaSwag | 10,042 | **27.38%** |
| ARC-Easy | 2,376 | **30.98%** |
| ARC-Challenge | 1,172 | **20.73%** |
| PIQA | 1,838 | **53.86%** |
| ArithMark-3 | 1,000 | **32.10%** |
| **Unweighted mean** | — | **33.01%** |

HellaSwag, ARC, and PIQA were evaluated with `lm-eval` 0.4.12 using exact continuation likelihood, zero few-shot examples, `acc_norm`, and the full task splits. ArithMark-3 used the official independent-tokenization continuation protocol and `acc_norm`. Raw result files are included under `evals/`.

**Takeaway:** Nero XS is meaningfully above random baselines on several tasks despite storing fewer than three million parameters. It remains an extremely small research model, and its benchmark differences should be interpreted with the reported sample sizes rather than as broad language competence.

## Architecture

| Component | Configuration |
|---|---|
| Architecture class | `NeroXSAForCausalLM` |
| Stored parameters | **2,996,480** |
| Physical / effective blocks | 10 / 14 |
| Recurrent layout | 1 prelude + 4 middle blocks × 2 passes + 5 coda |
| Residual width | 128 |
| Attention | 4 heads × 32 dimensions, full-rank Q/K/V/O |
| Attention modification | XSA value-subtraction after causal attention |
| MLP | Gated dense MLP, width 540 |
| Vocabulary | 2,048-token byte-level BPE |
| Embeddings | Tied input/output table |
| Position signal | Fixed sinusoidal coordinates, base 20,000 |
| Normalization | Learned centered unit normalization |

XSA removes from each attended head output its projection onto the current-token value vector. The four middle physical blocks are applied twice, providing fourteen effective block transformations without storing a second set of recurrent weights.

## Training story

### 1. DCLM base stage

- Exact stage budget: **4,999,999,488 tokens**.
- Sequence length 256, batch size 16, seed 7.
- AdamW: peak learning rate `0.012`, 10M-token warmup, cosine decay to `0.0012`, weight decay 0.1, betas `(0.9, 0.95)`, epsilon `1e-8`.
- Source-disjoint train/dev/test token streams produced from the local DCLM preparation pipeline.

### 2. FinePhrase continued pre-training

- Exact stage budget: **4,999,999,488 tokens**.
- Balanced equally across FinePhrase `faq`, `math`, `table`, and `tutorial` configurations during corpus preparation.
- Used only `rollout_results[0].text`; the source `text` field was ignored.
- Peak learning rate `0.0012`, no warmup, cosine decay to `0.00012`.
- Preserved the base model and complete AdamW optimizer state; reset only the dataset sampler and stage token counter.

Total recorded training exposure across both stages is **9,999,998,976 tokens**.

## Data lineage

| Stage | Source | Prepared training stream | Provenance |
|---|---|---:|---|
| Base | DCLM-derived local corpus | 814,270,415 unique prepared tokens, sampled repeatedly to the stage budget | Source JSONL SHA-256 `ed0ac03f1c4383ee3607271e5ae6a150f4bbc8dd5fcd598038eaee72134d09d8` |
| CPT | `HuggingFaceFW/finephrase` revision `78cf4a5ed0099214979c094c963e699c19163838` | 495,064,920 unique prepared train tokens, sampled repeatedly to the stage budget | 125M accepted tokens targeted per configuration; source-ID-disjoint split |

The FinePhrase preparation accepted 886,182 documents across the four configurations before assigning source-disjoint train/dev/test splits. Dataset sampling beyond each prepared stream's unique size means the training-token total is exposure, not unique-token count.

## Evaluation details

| Item | Value |
|---|---|
| Released checkpoint | FinePhrase stage `checkpoint_004999999488` |
| `lm-eval` version | 0.4.12 |
| Few-shot examples | 0 |
| Multiple-choice metric | `acc_norm` |
| Evaluation maximum context | 2,048 tokens (`lm-eval`); 1,024 for ArithMark-3 |
| Batch size | 16 (`lm-eval`); 32 examples for ArithMark-3 |
| Random seeds | lm-eval defaults recorded in the raw result file |

The released checkpoint was chosen by the highest unweighted mean across the five reported public tasks among the already evaluated training checkpoints. This is a benchmark-informed release selection, so the table is descriptive rather than an untouched final estimate.

## Reproducibility

| Item | Value |
|---|---|
| Framework | MLX 0.32.0 |
| Python during training | 3.14.6 |
| Hardware | Apple M5 Max |
| Training seed | 7 |
| Validation seed | 20260904 |
| DCLM manifest SHA-256 | `78a68559354521c2929678395a26740487e53a0648d8590123e2e655a2c562db` |
| FinePhrase manifest SHA-256 | `706c05ca84c0d6974c21f127b56076af8b56dd61748613885e4a084af8f75b0c` |
| Tokenizer SHA-256 | `5da5a6a51f165486e3e3ed9371bab510374d3d478ff105528e87fd8cfecd2705` |
| Released NPZ SHA-256 | `d49851109e6b087aabe6f484fe2dbb4f2e4ffa90276d9d11cc10938dd9234c7d` |
| Released safetensors SHA-256 | `fc67d9640b74a75eaadcad29edb40232692e066ced2cc5b511f276379e5fcfd3` |

`model.npz` and `model.safetensors` contain the same 103 FP32 tensors. The standalone release implementation was checked against the training implementation with exact logit equality (`max_abs_diff = 0.0`) on a fixed input.

## Limitations

- Fewer than three million parameters sharply limits factual recall, instruction following, coherence, and long-form generation.
- Training used 256-token sequences. Longer contexts are accepted by the deterministic coordinate implementation but are not established training-distribution behavior.
- The 2K tokenizer favors parameter efficiency over tokenization efficiency and has limited dedicated multi-digit coverage.
- Generation uses full-prefix recomputation and has no KV cache.
- FinePhrase content is synthetic and can transfer teacher errors or style.
- No independent human evaluation, safety evaluation, or comprehensive benchmark-contamination audit is published.
- Public benchmarks influenced checkpoint selection; do not treat these five scores as an untouched test set.
- This base model is not instruction-tuned and should not be expected to answer prompts like a chat assistant.

## Files

| File | Description |
|---|---|
| `model.safetensors` | FP32 weights in safetensors format |
| `model.npz` | Original MLX checkpoint weights |
| `modeling_nero_xs_mlx.py` | Standalone architecture, loading, and generation code |
| `config.json` | Exact released architecture configuration |
| `tokenizer.json`, `tokenizer_config.json` | 2K byte-level BPE tokenizer |
| `tokenizer_manifest.json` | Tokenizer construction provenance |
| `training_state.json` | Final checkpoint state and manifest identifiers |
| `evals/` | Raw full benchmark outputs |

## Citation

```bibtex
@misc{neroxs2026,
  author       = {j0no12},
  title        = {Nero XS: A 2.996M-Parameter Recurrent XSA Language Model},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/j0no12/Nero-XS}},
  note         = {Trained from scratch with MLX on Apple Silicon}
}
```

## License

The original Nero XS model weights and repository material are released under CC-BY-4.0. Attribution is required. The DCLM source is recorded as CC-BY-4.0, while FinePhrase is ODC-BY; those upstream dataset terms and attribution requirements remain applicable to their respective source material.