--- license: apache-2.0 language: - en tags: - text-generation - small-models - base-model - mla - jepa - experimental pipeline_tag: text-generation library_name: transformers datasets: - HuggingFaceFW/fineweb-edu ---

# Escarda-86M-Base ## 🔭 Jet-Long context extension (native 4K → 10K) This is the **Jet-Long** edition of [`Escarda-86M-Base`](https://huggingface.co/Quazim0t0/Escarda-86M-Base). It extends the usable context from the native **4,096**-token training window to **10,240** tokens with **no fine-tuning** and **no change to short-context behaviour**, by adding *dynamic bifocal RoPE* from **Jet-Long** ([arXiv:2607.07740](https://arxiv.org/abs/2607.07740), NVIDIA). ### What was applied Jet-Long pairs a **local window** (`w0 = 2048`, classic RoPE) with a **remote window** whose position map aliases far-apart tokens back onto the pretrained rotation grid: ``` f(x) = floor(x / G), G = max(1, ceil(L / 4096)) ``` `G` adapts to the **current** sequence length `L`, so: - **`L ≤ 4096` → `G = 1` → f is the identity → the model is bit-for-bit the base model.** (Verified: max |Δlogit| between Jet-Long on/off within the window is `0.000e+00`.) - **`L > 4096`** → the remote window keeps every rotation in-distribution, so the model extrapolates instead of collapsing. Implementation notes specific to this SpikeWhaleLM build: - Only the **decoupled RoPE partition** (16 of 64 head dims) is aliased; the NoPE partition is untouched. Softmax attention (`use_derf=False`) — the standard Jet-Long merge applies. - The remote view is realized by an **on-the-fly correction rotation** on the already-RoPE'd KV cache (RoPE composes additively), so **the cache is never rewritten** and decode is cheap. - Enabled via config: `use_jetlong=true`, `jetlong_w0=2048`, `jetlong_w_pretrained=4096`, `max_position_embeddings=10240`. Set `use_jetlong=false` to recover the exact base model. - The inclusion–exclusion / CuTe throughput kernel from the paper is **not** included (it targets 100K+ contexts on H100); at 86M params the bifocal attention is computed directly. ### Measured (PG-19-style perplexity on held-out text, lower is better) | Context length | Base model | This Jet-Long model | |---|---|---| | ≤ 4,096 (in-window) | *(identical — Jet-Long is a no-op)* | *(identical)* | | **10,240** | **59.68** | **16.04** | Beyond the training window the base model's perplexity blows up while Jet-Long stays flat — and long-context generation stays grammatical where the base model degrades into word-salad. ### Usage Jet-Long is **on by default** in this repo. Pass explicit `position_ids` so RoPE gets true absolute positions during cached decode: ```python import torch from transformers import AutoModelForCausalLM m = AutoModelForCausalLM.from_pretrained("Quazim0t0/Escarda-86M-Base-JL", trust_remote_code=True) ids = ... # up to ~10,240 tokens pos = torch.arange(ids.shape[1]).unsqueeze(0) out = m(input_ids=ids, position_ids=pos, use_cache=True) # prefill, then decode step-by-step ``` *Method: Tang, Wang, Gu, Han, Cai — “Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE”, arXiv:2607.07740. Applied here zero-shot to SpikeWhaleLM; no weights were retrained.* **Escarda-86M-Base** is a ~86M-parameter, from-scratch decoder-only language model — the **base** sibling of [Quazim0t0/Escarda-86M](https://huggingface.co/Quazim0t0/Escarda-86M) (the chat-tuned model). It shares the same `SpikeWhaleLM` architecture (Multi-head Latent Attention, an n-gram "engram" memory, hash-lookup layers, hyper-connections, an HRM refinement step, and JEPA / multi-token-prediction training objectives) and the same custom ChatML-aware tokenizer. This checkpoint is a JEPA-distilled base. It is best used as a **starting point for continued pretraining / fine-tuning** rather than as a chat assistant. > **Related models:** SFT / chat model → [Quazim0t0/Escarda-86M](https://huggingface.co/Quazim0t0/Escarda-86M) > · live demo → [Escarda-86M-Chat Space](https://huggingface.co/spaces/Quazim0t0/Escarda-86M-Chat) Trained using **Modal's credits** during the **Small Models, Big Adventures Hackathon**. --- ## Model summary | | | |---|---| | **Parameters** | ~85.7M (`tie_word_embeddings=True`) | | **Type** | Decoder-only LM (`SpikeWhaleLM`, `model_type: spike_whale`) | | **Hidden size / layers** | 640 / 16 | | **Attention** | 10 heads (`head_dim=64`), 1 KV head (MQA), MLA low-rank Q/O, decoupled RoPE(16)+NoPE(48), QK-norm | | **Context length** | 4096 tokens | | **Vocab** | 16,512 (custom `length-max` tokenizer) | | **License** | Apache-2.0 | For the full architecture description see the [chat model's card](https://huggingface.co/Quazim0t0/Escarda-86M#architecture). --- ## Architecture These models are built on **SpikeWhaleLM**, a custom ~86M-parameter decoder-only transformer (16 layers, hidden size 640, 4096-token context, 16,512 vocab, tied input/output embeddings). It combines several non-standard components: - **Multi-head Latent Attention (MLA + XSA)** — queries and the output projection are LoRA-compressed (rank 128); each head splits into a decoupled RoPE part (dim 16) and a position-agnostic NoPE part (dim 48); 10 query heads share a **single KV head** (multi-query attention), with QK-norm for stable logits. - **Engram n-gram memory** — a gated associative memory that hashes local n-grams (up to trigrams) into a learned 4,096-entry table and mixes the result back into the residual stream. - **Hash-lookup layers (×2)** — multi-head content-addressable features alongside the token embeddings. - **Hyper-Connections** — learned, width-expanded residual connections mixed via Sinkhorn-normalized routing, in place of the plain residual add. - **HRM refinement** — a Hierarchical Reasoning Model block that performs an extra latent "think a bit more" refinement pass over the hidden states before the output head. - **Multi-Token Prediction (MTP)** — a DeepSeek-V3-style auxiliary training head predicting more than one next token (no inference cost). - Feed-forward is **dense** (the block is MoE-capable, but MoE is disabled in this release). > **JEPA vs HRM.** The **Escarda** models are trained with **both HRM refinement and a JEPA (Joint-Embedding Predictive Architecture) auxiliary objective** (`use_hrm_refine=True`, `use_jepa=True`) — the JEPA term predicts future latent states during training to shape the model's representations. The sibling **Byrne** models drop JEPA and use **HRM refinement only**.

Architecture graph for Quazim0t0/Escarda-86M-Base. Open in hfviewer

## Tokenizer These models use **`SpikeTokenizer`**, a custom **byte-level "length-max" (greedy longest-match)** tokenizer with a **16,512-token vocabulary** — not a standard BPE/HF tokenizer. Text is UTF-8 encoded, each byte mapped to a latin-1 character, then greedily matched against the vocab using the longest key that fits at each position. It is **ChatML-aware**, with atomic special tokens for framing and reasoning/tool markers (`<|im_start|>`, `<|im_end|>`, ``/``, ``/``, tool-call markers) plus ``/``/``/``. It ships as a `PreTrainedTokenizer` subclass (`spike_tokenizer.py`) and loads via `AutoTokenizer.from_pretrained(..., trust_remote_code=True)`. ## Evaluation splits. byte_ppl is `exp(sum_NLL_nats / total_UTF8_bytes)` on WikiText-2 test (tokenizer- independent). BLiMP is fraction of minimal pairs with `logprob(good) > logprob(bad)` (12 paradigms × 150). Stderr is binomial `sqrt(p(1-p)/n)`. ### Language modeling | Metric | Value | |---|---| | WikiText-2 byte_ppl ↓ | **2.2228** | | BLiMP acc ↑ | 0.7144 | ### Multiple-choice suite | Task | acc | ± | acc_norm | ± | |---|---|---|---|---| | arc_easy | 0.3801 | 0.0100 | 0.3615 | 0.0099 | | arc_challenge | 0.1886 | 0.0114 | 0.2235 | 0.0122 | | hellaswag | 0.2759 | 0.0045 | 0.2832 | 0.0045 | | winogrande | 0.5162 | 0.0140 | — | — | | piqa | 0.5843 | 0.0115 | 0.5631 | 0.0116 | | openbookqa | 0.1300 | 0.0150 | 0.2500 | 0.0194 | | boolq | 0.5138 | 0.0087 | — | — | ### ArithMark-2.0 ([AxiomicLabs](https://huggingface.co/datasets/AxiomicLabs/ArithMark-2.0)) | Metric | Value | |---|---| | acc | 0.2536 ± 0.0087 | | acc_norm | 0.2348 ± 0.0085 | n = 2,500 · chance = 0.25. > **Note:** as a distilled base, this checkpoint has the **lowest byte-perplexity** of the > Escarda family but trades off downstream task accuracy — a good reminder that perplexity > alone is not a reliable capability ranking. For the strongest chat behaviour use > [Escarda-86M](https://huggingface.co/Quazim0t0/Escarda-86M); use **this** model when you > want a low-loss base to continue pretraining or fine-tune. --- ## Training & token budget - **Tokens:** ~20B (from-scratch pretraining of the SpikeWhale base, ~28k steps); this checkpoint is a JEPA-distilled snapshot of that base. - **Token/param ratio:** ~233 tokens/param (20B / 85.7M) — roughly **11–12× the Chinchilla ~20-tokens/param compute-optimal heuristic**, i.e. a deliberately **over-trained small model** (the inference-efficient trade-off). Fitting the Chinchilla data term to this model's own pretraining loss curve gives: `L(D) ≈ 2.611 + 77,715 · D^(−0.537)` (nats/token, R² = 0.92) From that fit: - **Compute-optimal tokens for this 86M size ≈ 4.3B** → the 20B run is **~4.6× past compute-optimal**. - **Diminishing-returns knee ≈ 22.5B** tokens (where +1B tokens buys < 0.005 nats) — the 20B stopping point lands **right at the knee**, a well-judged budget. - The model is **parameter-bound, not data-bound** at 20B: the capacity term (~0.82 nats) exceeds the data term (~0.54), so extra tokens help little. Doubling to 40B is projected to lower loss only ~0.07 nats (~7% perplexity) with negligible downstream gain — the lever for better quality is **more parameters, not more tokens**. (This is also why, as a distilled base, it reaches the lowest perplexity of the family without the best downstream scores — it is already at its data-term floor.) *Caveats: single-size fit (folds irreducible loss + capacity floor into one constant); the cosine-LR decay inflates the fitted exponent, so treat β as an upper bound; token counts are anchored to the ~20B figure and scale linearly if that differs.* --- ## Usage Custom architecture — load with `trust_remote_code=True` (the modeling code ships in this repo via `auto_map`): ```python from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained( "Quazim0t0/Escarda-86M-Base", trust_remote_code=True) ``` The tokenizer is the custom `SpikeTokenizer` (`tokenizer.json`, `algorithm: length-max`); load it with the `spike_tokenizer.py` helper from the project rather than `AutoTokenizer`. ## Acknowledgements Built with **Modal** credits during the **Small Models, Big Adventures Hackathon**, and released to the community as a base to build on. ## Citation If you use this model, please cite: ```bibtex @misc{escarda86mbase, title = {Escarda-86M-Base: A ~86M-parameter SpikeWhaleLM}, author = {Dean Byrne (Quazim0t0)}, year = {2026}, howpublished = {HuggingFace, \url{https://huggingface.co/Quazim0t0/Escarda-86M-Base}}, note = {Quazim0t0/Escarda-86M-Base} } ``` ## Escarda vs Byrne — vision family comparison The **Byrne** family uses HRM refinement. **Escarda** = Byrne **+ JEPA** (Joint-Embedding Predictive head) added *alongside* HRM in both the vision encoder and the LM trunk — auxiliary only, **zero inference cost**. **Vision encoder (DINOv2 teacher-alignment, n=1024 held-out):** | | Byrne-VE | Escarda-VE | |---|---|---| | Params | 39.34M | 39.60M (+JEPA head) | | CLS cosine | **0.776** | 0.771 | | PATCH cosine | **0.600** | 0.584 | | JEPA self-consistency | — | 0.040 | **Docling (same held-out doc images, atomic DocTags):** both emit well-formed DocTags; Byrne-Docling is marginally more complete on the hardest samples (closes ``, includes the `

` wrapper), consistent with its slightly higher teacher-alignment.
Escarda-Docling is structurally on par and adds the JEPA representation-learning trait.

**Pros/cons.** *Byrne (HRM):* higher teacher-alignment, all capacity on distillation
fidelity; no self-supervised objective. *Escarda (HRM+JEPA):* self-supervised
neighbour-prediction (richer spatial structure) at zero inference cost, trading ~1–3%
teacher-alignment. Same size class.

Family repos: [Byrne-VE](https://huggingface.co/Quazim0t0/Byrne-VE) ·
[Escarda-VE](https://huggingface.co/Quazim0t0/Escarda-VE) ·
[Byrne-Docling-131M](https://huggingface.co/Quazim0t0/Byrne-Docling-131M) ·
[Escarda-Docling-126M](https://huggingface.co/Quazim0t0/Escarda-Docling-126M)

## Update: engram repair (behavior-preserving)

The n-gram Engram memory in the original weights was degenerate: with the frozen
LSH compressor at init scale, every token hashed to bucket 0, so only one table
row ever received gradient. This revision rescales the (frozen) compressor and
broadcasts the learned bucket-0 vector across all table rows.

**Outputs are bit-identical to the previous revision** (verified: max logit
difference 0.0 across a prompt battery). The only change: the Engram's hash now
spreads across the full table and every bucket is independently trainable — so
if you distill or SFT on top of this base, the n-gram memory will actually learn
instead of staying a constant bias.