README.md

---
language:
- en
license: mit
library_name: transformers
tags:
- maba
- maba-v1.5
- recurrent
- dgda
- linear-attention
- sparse-attention
- maba-sa
- mla
- nope
pipeline_tag: text-generation
---

# Maba v1.5 (103.5M) Trained Checkpoint

Pretrained checkpoint of the **Maba v1.5 Experimental Architecture** trained on 3,044 dialogue pairs on NVIDIA L4 (bfloat16).

* Base Architecture Specification: [AndrewThompson1233/maba-v1.5-exp-architecture](https://huggingface.co/AndrewThompson1233/maba-v1.5-exp-architecture)
* Parameters: **103,520,911 (103.5M)**
* Core Computation Ratio: **95.21%** (4.30% Vocab Tax)
* Macro-Stack: **3:1** (15 DGDA Recurrence : 5 MABA-SA Dynamic Sparse Attention)
* Positional Encoding: **Strict NoPE** (0 positional parameters)

---

## Empirical Benchmark vs Qwen3.8-Flash-Next (101.7M)

Evaluated under identical training budgets (3,044 dialogues, 15 epochs, bfloat16, NVIDIA L4):

| Metric | Maba v1.5-exp | Qwen3.8-Flash-Next | Delta / Advantage |
| :--- | :---: | :---: | :---: |
| **Parameters** | **103,520,911 (103.5M)** | 101,701,120 (101.7M) | 0.2% parity |
| **Architecture** | **75% DGDA + 25% MABA-SA** | 75% GDN + 25% QSA + MoE | Cyclic 3:1 |
| **Positional Encoding** | **Strict NoPE (0 params)** | 25% Partial RoPE | Zero positional overhead |
| **Contrastive Retrieval (MCQ)** | **87.5% (7/8)** | 75.0% (6/8) | **+12.5% accuracy** |
| **Validation Loss** | **0.0697** | 0.0778 | **-10.4% entropy** |
| **Validation Perplexity (PPL)** | **1.07** | 1.08 | **Maba wins** |
| **Decode Throughput (L4)** | **7.0 tok/s** | 5.5 tok/s | **+27.3% faster generation** |

<p align="center">
  <img src="assets/empirical_benchmark.svg" width="920" alt="Maba v1.5 Empirical Benchmark" />
</p>

---

## Attention Ablation Proof

Empirical demonstration of the contribution of the 25% MABA-SA dynamic sparse attention layers against a pure linear recurrent baseline on the exact same checkpoint weights:

| Model Variant | Attention Mechanism | Validation Loss | Perplexity (PPL) | Error Reduction |
| :--- | :---: | :---: | :---: | :---: |
| **Pure DGDA (Ablation)** | None (100% Linear Recurrence) | 3.9360 | 51.21 | Baseline |
| **Qwen3.8-Flash-Next** | QSA (GQA + Micro-block Indexer) | 3.8772 | 48.29 | -5.7% vs Recurrence |
| **Maba v1.5 Full** | **MABA-SA (MLA + Top-32 + HCA)** | **3.5903** | **36.24** | **-29.2% error drop** |

---

## Needle-In-A-Haystack & Centroid Retrieval (512 to 4096 Tokens)

| Context Length | Needle Position | Needle Block | DG-Indexer (Hybrid Mean+Max) | Standard Pure Mean Pooling |
| :---: | :---: | :---: | :---: | :---: |
| **512 tokens** | 51 (10%) | Block #0 | **Retrieved (Top-32)** | Retrieved |
| **512 tokens** | 256 (50%) | Block #4 | **Retrieved (Top-32)** | Retrieved |
| **512 tokens** | 460 (90%) | Block #7 | **Retrieved (Top-32)** | Retrieved |
| **1024 tokens** | 102 (10%) | Block #1 | **Retrieved (Top-32)** | Retrieved |
| **1024 tokens** | 512 (50%) | Block #8 | **Retrieved (Top-32)** | Retrieved |
| **1024 tokens** | 921 (90%) | Block #14 | **Retrieved (Top-32)** | Retrieved |
| **2048 tokens** | 204 (10%) | Block #3 | **Retrieved (Top-32)** | Retrieved |
| **2048 tokens** | 1024 (50%) | Block #16 | **Retrieved (Top-32)** | Retrieved |
| **2048 tokens** | 1843 (90%) | Block #28 | **Retrieved (Top-32)** | Retrieved |
| **4096 tokens** | 2048 (50% Lost-in-Middle) | Block #32 | **Retrieved (Top-32)** | **Diluted to 0.0 (Failed)** |
| **4096 tokens** | 3686 (90%) | Block #57 | **Retrieved (Top-32)** | Retrieved |

---

## KV-Cache Footprint at 4k Context

| Context Length | Dense Attention (Baseline) | Qwen3.8-Flash-Next | Maba v1.5 (MLA + Top-32) | Memory Reduction vs Dense |
| :---: | :---: | :---: | :---: | :---: |
| **512 tokens** | 25.00 MB | 1.00 MB | **0.62 MB** | **-97.5%** |
| **1,024 tokens** | 50.00 MB | 2.00 MB | **1.25 MB** | **-97.5%** |
| **2,048 tokens** | 100.00 MB | 4.00 MB | **2.50 MB** | **-97.5%** |
| **4,096 tokens** | 200.00 MB | 8.00 MB | **2.50 MB** | **-98.8%** |