File size: 5,744 Bytes
336ba04 f1f2c96 336ba04 b0169a3 336ba04 f1f2c96 336ba04 | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 | ---
library_name: transformers
license: apache-2.0
language:
- multilingual
pipeline_tag: feature-extraction
tags:
- multimodal
- document-understanding
- masked-language-modeling
- long-context
---
<p align="left">
<img src="https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/assets/neomme/neomme_logo.webp?raw=true" alt="NeoMME logo" style="max-height: 140px;">
</p>
# NeoMME (800M): Single-Tower Multimodal-Native Multilingual Foundation Encoder
[](https://huggingface.co/docs/transformers/en/model_doc/neomme)
[](https://hf.co/collections/Hcompany/neomme)
[](https://arxiv.org/abs/2609.01657)
## Model summary
NeoMME is an efficient Multilingual and Multimodal-native foundational Encoder. Text tokens and raw image patches pass through one shared Transformer encoder. NeoMME does not use a separately pretrained vision encoder or a causal language model.
NeoMME-800M is a pretrained encoder backbone and cannot be used on its own for a downstream task. It returns contextual token representations, so users should fine-tune a task-specific head for retrieval, classification, extraction, or another downstream task. For document retrieval, use [NeoMME-800M-Retriever](https://huggingface.co/Hcompany/NeoMME-800M-Retriever).
<table>
<thead>
<tr style="background-color: rgba(146, 81, 247, 0.20);"><th>Specification</th><th>Value</th></tr>
</thead>
<tbody>
<tr><td>Parameters</td><td>800M</td></tr>
<tr><td>Vocabulary</td><td>131,072 tokens</td></tr>
<tr><td>Context length</td><td>16,384 tokens</td></tr>
<tr><td>Hidden size</td><td>1,792</td></tr>
<tr><td>Image patches</td><td>32 × 32 pixels, up to 2,048 pixels on the longest side (default)</td></tr>
</tbody>
</table>
## Usage
Use NeoMME with `transformers`:
```bash
# accelerate is an optional dependency needed only when using device_map="auto".
pip install -U accelerate transformers
```
The example below generates hidden states for a text document and a document image in one forward pass. The hidden states are not usable as is for a downstream task. If you are looking for retrieval embeddings, you should use the [NeoMME-800M-Retriever](https://huggingface.co/Hcompany/NeoMME-800M-Retriever) model instead.
**Generate hidden states**
```python
import requests
import torch
from PIL import Image
from transformers import AutoModel, AutoProcessor
def encode_document_text(processor, text: str) -> str:
return f"{processor.tokenizer.document_token}{text}"
model_id = "Hcompany/NeoMME-800M"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, device_map="auto")
text = "The cat sat on a mat."
image_url = "https://github.com/tonywu71/colpali-cookbooks/blob/main/examples/data/shift_kazakhstan.jpg?raw=true"
image = Image.open(requests.get(image_url, stream=True).raw)
inputs = processor(
text=[
encode_document_text(processor, text),
encode_document_text(processor, processor.image_token),
],
images=[image],
padding=True,
return_tensors="pt",
).to(model.device)
with torch.inference_mode():
outputs = model(**inputs)
text_hidden_states, image_hidden_states = outputs.last_hidden_state
```
NeoMME was pretrained with a masked discrete-diffusion objective. Therefore, the model can restore masked text tokens given the surrounding text (and, when present, image patches). The example below fills a single mask as a sanity check of that objective. It is not a generative or conversational model.
**Masked language modeling**
```python
import torch
from transformers import AutoModelForMaskedLM, AutoProcessor
model_id = "Hcompany/NeoMME-800M"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id, device_map="auto")
# Equivalent: "<doc>The capital of <mask> is London."
text = f"{processor.tokenizer.document_token}The capital of {processor.tokenizer.mask_token} is London."
inputs = processor(text=[text], return_tensors="pt").to(model.device)
with torch.inference_mode():
outputs = model(**inputs)
masked_index = (inputs.input_ids[0] == processor.tokenizer.mask_token_id).nonzero().item()
predicted_token_id = outputs.logits[0, masked_index].argmax(dim=-1)
print(processor.tokenizer.decode(predicted_token_id))
```
## Training
NeoMME-800M was pretrained from scratch on multilingual text and visual-text data, including web text, code, math, document pages, captions, and synthetic OCR data. The model learns to restore masked text tokens. For document images paired with transcripts, image patches remain visible and the pretraining objective has no pixel reconstruction loss.
The [NeoMME technical report](https://arxiv.org/abs/2609.01657) describes the full pretraining recipe.
## Limitations
- NeoMME-800M is a pretrained encoder backbone and requires task-specific fine-tuning.
- The model has not received a comprehensive safety, bias, or privacy evaluation.
## License
Model weights are released under the Apache 2.0 license.
## Citation
```bibtex
@misc{lac2026neommesingletowermultimodalnativemultilingual,
title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference},
author={Aurélien Lac and Tony Wu},
year={2026},
eprint={2609.01657},
archivePrefix={arXiv},
primaryClass={cs.IR},
url={https://arxiv.org/abs/2609.01657},
}
```
|