M.A.T.E.R.I.A. V4 1B
Multi-Analytical Toroidal Engine for Recursive Intelligent Analysis
JEPA-First Toroidal Architecture with HSAQ Adaptive Optimization
[](https://huggingface.co/MethodWhite/materia-v4-1b-bpe)
[](https://huggingface.co/MethodWhite/materia-v4-1b-bpe)
[](https://huggingface.co/MethodWhite/materia-v4-1b-bpe/blob/main/LICENSE)
[](https://huggingface.co/MethodWhite/materia-v4-1b-bpe/blob/main/materia-v4.Q4_0.gguf)
[](https://huggingface.co/MethodWhite/materia-v4-1b-bpe)
---
## π Model Description
**M.A.T.E.R.I.A. V4** (Multi-Analytical Toroidal Engine for Recursive Intelligent Analysis) is a novel neural architecture that departs from the traditional linear transformer paradigm. Instead of processing tokens through a sequential stack, MATERIA employs a **toroidal (donut-shaped) geometry** where GQA, SNN, and SSM components converge into a unified **JEPA latent space** through hexagonal interconnection.
The key architectural insight is that information flows in **cycles** through a toroidal topology, with each cycle inheriting sparse activation patterns from the previous one (70/30 blend). This creates a recurrent processing structure without the computational overhead of traditional recurrence.
### Architecture Overview
```
βββββββββββββββββββββ
/ Transformer \
| β β β |
βββββββββ | β JEPA β | βββββββββ
β SSM βββββ β ββββ | β SNN β
βββββββββ | β Hub β | βββββββββ
| β β β |
\ /
ββββββββββ¬ββββββββββ
β
ββββββββ΄βββββββ
β Head/Emb β
βββββββββββββββ
```
### Components
| Component | Role | Specification |
|-----------|------|---------------|
| **JEPA Hub** | Central latent space predictor | SCA spectral decomposition, Ξ»β = KΒ·Ο(ΞΌβ), K=2.781042 |
| **GQA** | Grouped Query Attention | 24 heads, 6 KV heads, RoPE, Flash Attention 2 |
| **SNN** | Spiking Neural Network | LIF neurons, Ο=0.8, surrogate gradient (Ξ²=5.0), 30% target rate |
| **SSM** | State Space Model | 64-state, long-range dependency capture |
| **HSAQ** | HyperSparse Adaptive Quantization | 10 per-edge learnable instances, STE, kthvalue dynamic threshold |
| **Toroidal** | 3-cycle recurrence | Mask inheritance 70/30, hexagonal interconnect |
## π¬ Key Innovation: HSAQ Optimizer
**HSAQ (HyperSparse Adaptive Quantization)** is the core innovation β it **replaces AdamW** as the model optimizer while simultaneously providing adaptive sparse activation.
Unlike traditional optimizers that maintain complex state (AdamW: 2 states/param, 8 bytes/param), HSAQ uses:
- **SGD Nesterov** (momentum=0.9): 1 state/param, 4 bytes/param (2.5Γ memory savings)
- **Per-edge learnable sparsity**: 10 `nn.Parameter(sparsity_logit)` instances
- **Dynamic kthvalue thresholding**: sparsity adapts per-batch based on activation distribution
- **Straight-Through Estimator (STE)**: forward hard mask, backward sigmoid relaxation β fully differentiable
- **Sacred geometry initialization**: logits from golden ratio principles
- **Toroidal mask inheritance**: 70/30 blend across cycles preserves activation topology
### HSAQ Sparsity Distribution
| Edge | Initial Logit | Target Sparsity | Role |
|------|--------------|-----------------|------|
| Embedding | -3.0 | ~5% | Token projection |
| JEPA in | -3.0 | ~5% | Hub entry |
| T2 | -2.5 | ~8% | Transformer layer 2 |
| T5 | -2.0 | ~12% | Transformer layer 5 |
| T8 | -1.7 | ~15% | Transformer layer 8 |
| T Cycle | -2.5 | ~8% | Toroidal recurrence |
| SNN | -3.5 | ~3% | Spiking (minimal, spikes already sparse) |
| SNN Latent | -3.0 | ~5% | SNNβJEPA projection |
| SSM | -3.0 | ~5% | State model |
| SSM Latent | -3.0 | ~5% | SSMβJEPA projection |
## βοΈ Training Configuration
### Model Hyperparameters
| Hyperparameter | Value |
|---------------|-------|
| Total Parameters | **1,335,302,155** (1.34B) |
| Architecture | Toroidal JEPA-First + SCA |
| Hidden Dimension | 1792 |
| Latent Dimension | 1792 |
| Layers | 24 |
| Attention Heads | 24 |
| KV Heads | 6 |
| SNN Dimension | 1792 |
| SSM State Size | 64 |
| Toroidal Cycles | 3 |
| Vocabulary | 32,768 (BPE) |
| Tokenizer | SentencePiece multilingual v2 |
### Training Hyperparameters
| Hyperparameter | Value |
|---------------|-------|
| Optimizer | SGD Nesterov (momentum=0.9) + HSAQ |
| Learning Rate | 1.5e-4 β cosine decay |
| Warmup Steps | 2,000 |
| Weight Decay | 0.1 |
| Gradient Clipping | 1.0 |
| Batch Size | 64 (effective) |
| Sequence Length | 128 |
| Gradient Accumulation | 32 |
| Mixed Precision | bf16 |
| Gradient Checkpointing | Enabled |
| Epochs | 2/10 |
| Training Duration | 327.7 min (5.46h) |
### Hardware
| Component | Specification |
|-----------|--------------|
| GPU | 1Γ NVIDIA RTX 6000 Ada (48GB) |
| VRAM Usage | 24GB / 48GB (stable) |
| CPU | 16 vCPUs |
| RAM | 70GB |
| Provider | Brev.dev / NVIDIA |
### Dataset
- **Primary**: [HuggingFaceFW/fineweb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) (CC-MAIN-2024-10)
- **Supplemental**: Wikipedia (12 languages: EN, ES, FR, DE, PT, AR, HI, JA, KO, RU, IT, ZH)
- **Size**: ~5M lines, 625,892 training chunks at seq_len=128
- **Format**: BPE-tokenized with SentencePiece multilingual model
## π Training Results
### Final Metrics
| Metric | Value | Notes |
|--------|-------|-------|
| **Total Loss** | **0.1303** | Dual loss: token + KΒ·jepa |
| **Token Loss** | ~8.06 | Cross-entropy |
| **JEPA MSE** | ~0.052 | Normalized by latent variance |
| **Accuracy** | **6.36%** | Token prediction (early training) |
| **Perplexity** | **~4186** | Expected for 2-epoch 1B model |
| **SNN Spike Rate** | **30.3%** | β
Perfectly calibrated to target |
| **HSAQ Sparsity** | 0.029 (target 0.048) | Adaptive, undershooting target |
| **NaN Events** | **0** | Stable training throughout |
### Performance Curves
| Metric | Trend |
|--------|-------|
| **Loss** | 8.9 β 0.13 (smooth decay) |
| **Accuracy** | 4.6% β 6.36% (improving) |
| **Perplexity** | 7,941 β 4,186 (declining) |
| **SNN Spike Rate** | 46% β 30% (perfectly regulated) |
| **HSAQ Sparsity** | 0.026 β 0.029 (stable adaptive) |
| **Learning Rate** | Linear warmup β cosine decay |
### Generation Samples (E2 Final)
| Prompt | Generation |
|--------|------------|
| "The meaning of life is" | `constitutional a p juan alum. i spect a recently` |
| "Artificial intelligence" | `. a mc. anstuff is the includingporter is youg sch` |
| "Hello, how are you" | `group and i for and at the new say and averaged an` |
*Note: Model shows emerging word structure and phrase formation at 2 epochs. Full linguistic capability requires 10+ epochs.*
## πΎ Files
| File | Size | Format | Description |
|------|------|--------|-------------|
| `materia-v4.basemateria` | 5.9 GB | Pickle (native) | Full model weights + config + tokenizer |
| `materia-v4.Q4_0.gguf` | 5.5 GB | GGUF | Q4_0 quantized (llama.cpp/Ollama/LM Studio) |
| `checkpoint_epoch1.pt` | 11.2 GB | PyTorch | Epoch 1 with optimizer state |
| `checkpoint_epoch2.pt` | 10.7 GB | PyTorch | Epoch 2 with optimizer state |
| `config_1B.yaml` | 839 B | YAML | Training configuration |
## π Usage
### PyTorch (Full Precision)
```python
import torch
import pickle
# Load .basemateria
with open('materia-v4.basemateria', 'rb') as f:
data = pickle.load(f)
state_dict = data['state_dict']
config = data['config']
print(f"Model: {data['config']['version']}")
print(f"Dimensions: {config['dim']}, Vocab: {config['vocab_size']}")
# Import model
from models.materia_v4 import MateriaV4
model = MateriaV4(
vocab_size=config['vocab_size'],
dim=config['dim'],
n_layers=24,
n_heads=24,
n_kv=6,
latent_dim=config['latent_dim'],
)
model.load_state_dict(state_dict, strict=False)
model.eval()
# Generate
import torch.nn.functional as F
stoi = data['tokenizer']
itos = {v: k for k, v in stoi.items()}
prompt = "The meaning of life is"
input_ids = torch.tensor([[stoi.get(c, 0) for c in prompt]])
with torch.no_grad():
for _ in range(50):
logits, _, _ = model(input_ids)
p = F.softmax(logits[:, -1, :] / 0.8, dim=-1)
next_id = torch.multinomial(p, 1)
input_ids = torch.cat([input_ids, next_id], dim=1)
output = ''.join([itos.get(i.item(), '
