Hugging Face's logo

state-spaces
/
mamba3-mimo-1.5b

MambaSSM
PyTorch
mamba
mamba-3
mimo
state-space-model
causal-lm
language-modeling
Model card Files
xet
 Community
mamba3-mimo-1.5b
File size: 2,500 Bytes
bc6b5d0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
---
license: apache-2.0
library_name: mamba-ssm
tags:
- mamba
- mamba-3
- mimo
- state-space-model
- causal-lm
- language-modeling
---

# Mamba-3 MIMO 1.50 B

`mamba3-mimo-1.5b` is a pretrained causal language model with 1.50 B
parameters. It is built from stacked blocks, each containing a Mamba-3
MIMO mixer followed by a gated MLP. It contains no attention layers and
is released with BF16 weights in the public `mamba_ssm` checkpoint format.

## Model architecture

| Property | Value |
|---|---:|
| Parameters | 1.50 B |
| Layers | 24 |
| Model dimension | 2,048 |
| SSM state size | 128 |
| SSM head dimension | 64 |
| SSM heads | 64 |
| SSM groups | 1 |
| MIMO rank | 4 |
| Chunk size | 16 |
| Context length | 2,048 |

The model was pretrained on 100B tokens from
[FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu).
It uses tied input and output embeddings and the `meta-llama/Llama-3.1-8B` tokenizer.

## Installation

Install CUDA-enabled PyTorch first, followed by the latest Mamba source:

```bash
pip install git+https://github.com/state-spaces/mamba.git --no-build-isolation
```

While this repository is private, authenticate with Hugging Face:

```bash
hf auth login
```

## Usage

```python
import torch
from transformers import AutoTokenizer
from mamba_ssm.models.mixer_seq_simple import MambaLMHeadModel

model_id = "state-spaces/mamba3-mimo-1.5b"

tokenizer = AutoTokenizer.from_pretrained(
    "meta-llama/Llama-3.1-8B",
)

model = MambaLMHeadModel.from_pretrained(
    model_id,
    device="cuda",
    dtype=torch.bfloat16,
)
model.eval()

input_ids = tokenizer(
    "Mamba-3 is",
    return_tensors="pt",
).input_ids.cuda()

with torch.inference_mode():
    logits = model(input_ids).logits

print(logits.shape)
```

## References

- [Mamba-3: Improved Sequence Modeling using State Space Principles](https://arxiv.org/abs/2603.15569)
- [Official Mamba implementation](https://github.com/state-spaces/mamba)

## Citation

If you use this model, please cite:

```bibtex
@misc{lahoti2026mamba3improvedsequencemodeling,
  title         = {Mamba-3: Improved Sequence Modeling using State Space Principles},
  author        = {Aakash Lahoti and Kevin Y. Li and Berlin Chen and
                   Caitlin Wang and Aviv Bick and J. Zico Kolter and
                   Tri Dao and Albert Gu},
  year          = {2026},
  eprint        = {2603.15569},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2603.15569}
}
```