--- license: apache-2.0 language: - en library_name: transformers pipeline_tag: text-generation datasets: - HuggingFaceFW/fineweb-edu - mlfoundations/dclm-baseline-1.0 tags: - tiny - small - pretrained-from-scratch - qwen3 - research co2_eq_emissions: emissions: 460 source: estimated from single-A100 training run training_type: pre-training hardware_used: 1x NVIDIA A100 ---

SupraNeo-4M

# SupraNeo-4M **A 4.07M-parameter decoder-only language model built to test a single question: how much of a tiny model should actually compute anything?** Most models at this scale spend the majority of their parameters on a vocabulary lookup table. SupraNeo-4M spends **82%** on the transformer stack. The design starts from a custom 4,096-token BPE tokenizer, which drops the embedding matrix to 524k parameters and frees the rest for a deep, narrow stack. The aspect ratio (d/L ≈ 13) is deliberately below what the 100M class uses — following the MobileLLM finding that optimal depth-to-width shifts toward depth as models shrink. ## Architecture | | | |---|---| | Parameters | **4,070,240** (3.41M non-embedding, 83.9%) | | Architecture | Qwen3 (`Qwen3ForCausalLM`) | | Hidden size | 160 | | Layers | 12 | | Attention heads | 4 (head_dim 40) | | KV heads | 2 (GQA 2:1) | | MLP intermediate | 432 (SwiGLU) | | Vocabulary | 4,096 (custom BPE, ~3.2 chars/token) | | Context length | 1,024 | | Normalization | RMSNorm + QK-Norm | | Embeddings | tied | | Precision | float32 | No custom modeling code — `trust_remote_code` is **not** required. ## Usage ```python from transformers import AutoModelForCausalLM, AutoTokenizer tok = AutoTokenizer.from_pretrained("DedeProGames/SupraNeo-4M") model = AutoModelForCausalLM.from_pretrained("DedeProGames/SupraNeo-4M") ids = tok("The main reason that", return_tensors="pt") out = model.generate(**ids, max_new_tokens=40, temperature=0.8, top_p=0.9, do_sample=True) print(tok.decode(out[0])) ``` This is a **base model** with no chat template. `apply_chat_template` will fail by design. ## Training Pre-trained from scratch on a **single NVIDIA L4** over **5B tokens** (~1,230 tokens per parameter). | | | |---|---| | Data | 84% [HuggingFaceFW/fineweb-edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) filtered to `int_score ≥ 4`, 16% [mlfoundations/dclm-baseline-1.0](https://huggingface.co/datasets/mlfoundations/dclm-baseline-1.0) | | Schedule | WSD, peak LR 4e-3, 1,500-step warmup, 1-sqrt decay over the final 20% | | Anneal | last 20% on the FineWeb-Edu subset only, context extended 512 → 1,024 | | Batch | 65,536 tokens/step | | Optimizer | AdamW (β 0.9/0.95, wd 0.1 on 2D params, grad clip 1.0), z-loss 1e-4 | | Init | residual branches scaled by 1/√(2L) | The heavy FineWeb-Edu weighting is deliberate. DCLM-baseline was tuned to win reasoning benchmarks in the 1–7B range, and none of that transfers at 4M — what a model this size can learn is register and local fluency, and FineWeb-Edu's uniform expository prose is far easier to model. ## Evaluation — BananaMind Base Bench 1.1 350 items, 4-way continuation-likelihood, `add_special_tokens=False`, no BOS, selection by highest conditional mean log-prob, Elo by weighted MLE with a prior of 4 games at 1000. **Official run**: dataset checksum verified, schema verified, 0 truncated contexts, 0 truncated continuations. | Category | Accuracy | z vs chance | Elo | Sig. | |---|---:|---:|---:|:--:| | Language Completion | **56.0%** | +5.06 | 963 | ✅ | | Logical Reasoning | 38.0% | +2.12 | 978 | ✅ | | World Knowledge | 34.0% | +1.47 | 820 | | | Context Tracking | 34.0% | +1.47 | 856 | | | Commonsense | 26.0% | +0.16 | 770 | | | Quantitative | 24.0% | −0.16 | 834 | | | Code Completion | 22.0% | −0.49 | 883 | | | | | |---|---| | **Overall Elo** | **868** | | Chance floor | 805 | | Above floor | **+63** | | Raw accuracy | 33.4% (95% CI 28.5–38.4%) | | z vs chance | **+3.64 — significant** | By difficulty: easy 39.3%, medium 29.1%, hard 31.9%. ### Reading these numbers honestly The aggregate is significantly above chance, but the signal is concentrated in one place. **Language Completion at 56% (+5.06σ) is the only strongly separated category**, and that is exactly what a 4M model should be able to do: local grammatical and register plausibility. Logical Reasoning clears the bar marginally. The remaining five categories sit within noise, and Quantitative and Code Completion land at or slightly below chance — this model has no arithmetic or code capability, and the card should not be read as claiming otherwise. The medium/hard inversion (29.1% vs 31.9%) is noise at n≈117, not evidence that harder items are easier. BananaMind Base Bench was calibrated for the 65M–100M+ range. At 4M, with 350 four-way items, the detection floor at 1.96σ is roughly 29.5% accuracy — most of this benchmark simply lacks resolution here. For tracking progress at this scale, **bits-per-byte** on held-out text and **BLiMP** are the metrics with actual sensitivity. HellaSwag, PIQA and ARC are not reported because they sit at chance and measure nothing. ## Limitations SupraNeo-4M produces grammatical, register-consistent English with coherence over one to two sentences and topical drift beyond that. It has no factual reliability, no arithmetic, no code ability, and no instruction following. The 4,096-token vocabulary means it compresses text ~35% less efficiently than a standard 32k tokenizer, and its outputs are not comparable to other models by raw cross-entropy — use bits-per-byte. This is a **research artifact** for studying small-scale pretraining, vocabulary budgets, and data mixtures. It is not intended for deployment. ## Carbon footprint Training was estimated to emit **0.46 kg CO₂ eq.** — a single L4 for the duration of the run. For reference, that is roughly the footprint of driving a passenger car about 2.5 kilometers. --- *by DedeProGames*