README.md

---
library_name: transformers
license: apache-2.0
language:
- multilingual
pipeline_tag: feature-extraction
tags:
- multimodal
- document-understanding
- masked-language-modeling
- long-context
---

<p align="left">
  <img src="https://github.com/tonywu71/colpali-cookbooks/blob/6ef1332da6bcb48c7ef1f19b25bfa555be7031a8/assets/neomme/neomme_logo.webp?raw=true" alt="NeoMME logo" style="max-height: 140px;">
</p>

# NeoMME (800M): Single-Tower Multimodal-Native Multilingual Foundation Encoder

[![Hugging Face](https://img.shields.io/badge/Model_doc-FFD21E?style=for-the-badge&logo=huggingface&logoColor=000)](https://huggingface.co/docs/transformers/en/model_doc/neomme)
[![Hugging Face](https://img.shields.io/badge/Collection-FFD21E?style=for-the-badge&logo=huggingface&logoColor=000)](https://hf.co/collections/Hcompany/neomme)
[![arXiv](https://img.shields.io/badge/arXiv-2609.01657-b31b1b.svg?style=for-the-badge)](https://arxiv.org/abs/2609.01657)

## Model summary

NeoMME is an efficient Multilingual and Multimodal-native foundational Encoder. Text tokens and raw image patches pass through one shared Transformer encoder. NeoMME does not use a separately pretrained vision encoder or a causal language model.

NeoMME-800M is a pretrained encoder backbone and cannot be used on its own for a downstream task. It returns contextual token representations, so users should fine-tune a task-specific head for retrieval, classification, extraction, or another downstream task. For document retrieval, use [NeoMME-800M-Retriever](https://huggingface.co/Hcompany/NeoMME-800M-Retriever).

<table>
<thead>
<tr style="background-color: rgba(146, 81, 247, 0.20);"><th>Specification</th><th>Value</th></tr>
</thead>
<tbody>
<tr><td>Parameters</td><td>800M</td></tr>
<tr><td>Vocabulary</td><td>131,072 tokens</td></tr>
<tr><td>Context length</td><td>16,384 tokens</td></tr>
<tr><td>Hidden size</td><td>1,792</td></tr>
<tr><td>Image patches</td><td>32 × 32 pixels, up to 2,048 pixels on the longest side (default)</td></tr>
</tbody>
</table>

## Usage

Use NeoMME with `transformers`:

```bash
# accelerate is an optional dependency needed only when using device_map="auto".
pip install -U accelerate transformers
```

The example below generates hidden states for a text document and a document image in one forward pass. The hidden states are not usable as is for a downstream task. If you are looking for retrieval embeddings, you should use the [NeoMME-800M-Retriever](https://huggingface.co/Hcompany/NeoMME-800M-Retriever) model instead.

**Generate hidden states**

```python
import requests
import torch
from PIL import Image

from transformers import AutoModel, AutoProcessor


def encode_document_text(processor, text: str) -> str:
    return f"{processor.tokenizer.document_token}{text}"


model_id = "Hcompany/NeoMME-800M"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, device_map="auto")

text = "The cat sat on a mat."
image_url = "https://github.com/tonywu71/colpali-cookbooks/blob/main/examples/data/shift_kazakhstan.jpg?raw=true"
image = Image.open(requests.get(image_url, stream=True).raw)

inputs = processor(
    text=[
        encode_document_text(processor, text),
        encode_document_text(processor, processor.image_token),
    ],
    images=[image],
    padding=True,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    outputs = model(**inputs)

text_hidden_states, image_hidden_states = outputs.last_hidden_state
```

NeoMME was pretrained with a masked discrete-diffusion objective. Therefore, the model can restore masked text tokens given the surrounding text (and, when present, image patches). The example below fills a single mask as a sanity check of that objective. It is not a generative or conversational model.

**Masked language modeling**

```python
import torch

from transformers import AutoModelForMaskedLM, AutoProcessor


model_id = "Hcompany/NeoMME-800M"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMaskedLM.from_pretrained(model_id, device_map="auto")

# Equivalent: "<doc>The capital of <mask> is London."
text = f"{processor.tokenizer.document_token}The capital of {processor.tokenizer.mask_token} is London."
inputs = processor(text=[text], return_tensors="pt").to(model.device)

with torch.inference_mode():
    outputs = model(**inputs)

masked_index = (inputs.input_ids[0] == processor.tokenizer.mask_token_id).nonzero().item()
predicted_token_id = outputs.logits[0, masked_index].argmax(dim=-1)
print(processor.tokenizer.decode(predicted_token_id))
```

## Training

NeoMME-800M was pretrained from scratch on multilingual text and visual-text data, including web text, code, math, document pages, captions, and synthetic OCR data. The model learns to restore masked text tokens. For document images paired with transcripts, image patches remain visible and the pretraining objective has no pixel reconstruction loss.

The [NeoMME technical report](https://arxiv.org/abs/2609.01657) describes the full pretraining recipe.

## Limitations

- NeoMME-800M is a pretrained encoder backbone and requires task-specific fine-tuning.
- The model has not received a comprehensive safety, bias, or privacy evaluation.

## License

Model weights are released under the Apache 2.0 license.

## Citation

```bibtex
@misc{lac2026neommesingletowermultimodalnativemultilingual,
      title={NeoMME: A Single-Tower Multimodal-Native Multilingual Foundation Encoder for Efficient Fine-Tuning and Inference},
      author={Aurélien Lac and Tony Wu},
      year={2026},
      eprint={2609.01657},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2609.01657},
}
```