File size: 7,169 Bytes
8cac147 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 | ---
language:
- en
license: other
license_name: jtrl-v1.0
license_link: LICENSE
base_model: dhanesh-hf/Jarvis-Titan-V15-MoE-Decoupled
tags:
- moe
- deepseek-moe
- titans-neural-memory
- tri-brid-memory
- differential-holographic-attention
- dha-3
- csa3-perturbative-attention
- multi-token-prediction
- reasoning
- math
- code
- agentic
- long-context
pipeline_tag: text-generation
library_name: transformers
---
<div align="center">
# 🏛️ J.A.R.V.I.S. TITAN 14.8B MoE (Milestone M4 — CSA3)
### Frontier High-Density Reasoning & Adaptive Memory Architecture
**14.8B DeepSeekMoE Backbone + M4 Tri-Brid Neural Memory + CSA3 Differential Attention + Multi-Token Speculative Prediction**
[](https://huggingface.co/dhanesh-hf/Jarvis-Titan-M4-MoE-CSA3)
[](https://huggingface.co/dhanesh-hf/Jarvis-Titan-V15-MoE-Decoupled)
[](https://arxiv.org/abs/2501.00663)
[](./LICENSE)
[]()
</div>
---
## ⚡ Overview
**J.A.R.V.I.S. Titan 14.8B MoE (CSA3)** is a premier high-density reasoning model engineered for complex mathematical derivation, algorithmic synthesis, and extended-context cognitive tasks.
Built upon an upcycled 14.8B DeepSeekMoE backbone, the model incorporates the calibrated **M4 Tri-Brid Neural Memory** system, **Continuous Sparse Attention 3 (CSA3)**, and **Multi-Token Prediction (MTP)**, preserving full mathematical precision while enabling efficient long-context associative recall.
---
## 🚀 Key Architectural Upgrades & Features
`Jarvis-Titan-M4-MoE-CSA3` introduces five major capability enhancements over conventional dense and sparse transformers:
### 1. 🧠 M4 Tri-Brid Neural Memory Subsystem
* **Multi-Tier Cognitive Storage**: Integrates high-throughput local sliding window attention, an exact salient needle-in-a-haystack reservoir, and associative test-time neural memory matrices ($M_t$).
* **Eliminates Associative Recall Decay**: Maintains robust long-horizon retrieval across extended token contexts (up to 131,072 tokens) without suffering from the context fading typical of standard linear recurrent systems.
### 2. ⚡ Continuous Sparse Attention 3 (CSA3)
* **Differential KV Caching**: Uses quantum perturbation principles to compress intermediate attention states into continuous differential representations.
* **Bounded Attention Footprint**: Significantly minimizes key-value memory overhead during long multi-turn sessions and complex chain-of-thought derivations.
### 3. 🎯 Length-Adaptive Dynamic Memory Gating
* **Zero-Distortion Short Prompt Reasoning**: Gating dynamics smoothly adapt according to sequence length, maintaining 100% local attention fidelity with zero degradation on short queries, STEM problems, and interactive coding.
* **Dynamic Recall Expansion**: Automatically scales neural memory bandwidth as sequence depth grows, ensuring immediate access to critical needle tokens in long documents.
### 4. ⏩ Speculative Multi-Token Prediction (MTP)
* **Parallel Target Projections**: Features native multi-token prediction heads to forecast subsequent tokens in parallel.
* **Accelerated Generation Throughput**: Enables speculative decoding and verification speedups of up to $1.8\times$ to $2.2\times$ during inference serving.
### 5. 🛡️ 100% Preserved 120M High-Density Reasoning Core
* Fully preserves the verified STEM mathematics, Olympiad problem-solving, and executable code synthesis intellect developed during the 120M token high-density DeepSeekMoE training curriculum.
---
## 📋 Model Architecture & Specifications
| Attribute | Specification |
| :--- | :--- |
| **Model Name** | J.A.R.V.I.S. Titan 14.8B MoE — CSA3 |
| **Base Architecture** | DeepSeekMoE Sparse Mixture-of-Experts |
| **Total Parameters** | 14.8 Billion |
| **Active Parameters** | ~3.2 Billion per token |
| **Total Layers** | 28 Transformer Layers |
| **Routing Topology** | 8 Routed Experts + 1 Isolated Shared Expert (Top-2 active) |
| **Attention Mechanism** | Grouped-Query Attention (GQA, 28 Q-Heads : 4 KV-Heads) |
| **Memory Enhancement** | M4 Tri-Brid Neural Memory (Sliding Window + Salient Reservoir + Neural Recurrence) |
| **Differential Attention** | Continuous Sparse Attention 3 (CSA3) |
| **Speculative Decoding** | Multi-Token Prediction (MTP) Head |
| **Vocabulary Size** | 152,064 tokens |
| **Max Context Length** | Up to 131,072 tokens |
| **Precision** | Float16 / Bfloat16 |
---
## 🛠️ Quickstart & Inference Guide
The model is compatible with Hugging Face `transformers` and can be loaded directly onto GPU/accelerator hardware:
```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "dhanesh-hf/Jarvis-Titan-M4-MoE-CSA3"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
model_id,
device_map="auto",
torch_dtype=torch.float16,
trust_remote_code=True
)
prompt = "<|im_start|>system\nYou are J.A.R.V.I.S., an expert reasoning assistant engineered by Dhanesh. Before answering, think through the problem carefully inside <think>...</think> tags. Verify your reasoning. If you detect a flaw, correct it. Provide your final solution inside <answer>...</answer> tags. When the problem asks for a boxed answer, use \\boxed{your answer}. Be concise but complete.<|im_end|>\n<|im_start|>user\nSolve for x: 3x + 15 = 42.<|im_end|>\n<|im_start|>assistant\n<think>\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
outputs = model.generate(
**inputs,
max_new_tokens=1024,
temperature=0.6,
top_p=0.9,
repetition_penalty=1.15
)
print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))
```
---
## 🛡️ License & Terms of Use
This model is governed by the **J.A.R.V.I.S. Titan Proprietary Research License (JTRL-v1.0)**.
* **Permitted:** Non-commercial academic audit, independent scientific evaluation, and benchmark replication.
* **Strictly Prohibited:** Commercial exploitation, hosted inference services/APIs, unauthorized weights redistribution, and competitive model distillation.
* Please consult the complete [`LICENSE`](./LICENSE) file for full terms and conditions.
### Citation
```bibtex
@misc{jarvis_titan_m4_csa3_2026,
author = {Dhanesh},
title = {J.A.R.V.I.S. Titan 14.8B MoE: Unified Tri-Brid Neural Memory and Autonomous Reasoning Architecture},
year = {2026},
publisher = {Hugging Face},
howpublished = {\url{https://huggingface.co/dhanesh-hf/Jarvis-Titan-M4-MoE-CSA3}}
}
```
|