| --- |
| title: MTEB-BR - Brazilian Portuguese (PT-BR) |
| emoji: 🏆 |
| colorFrom: green |
| colorTo: yellow |
| sdk: static |
| pinned: false |
| license: apache-2.0 |
| --- |
| |
| # MTEB-BR - Brazilian Portuguese (PT-BR) |
|
|
| A public benchmark for evaluating text embedding models on **Brazilian Portuguese**, built as a thin extension on top of the [`mteb`](https://github.com/embeddings-benchmark/mteb) library. |
|
|
| ## What you'll find here |
|
|
| - 🏆 **[Leaderboard](https://huggingface.co/spaces/MTEB-BR/leaderboard)** — interactive ranking, 93 models × 22 tasks, Pareto chart |
| - 📊 **[`mteb-pt-results`](https://huggingface.co/datasets/MTEB-BR/mteb-pt-results)** — all per-task JSONs + per-query parquets, ~1100 files |
| - 💻 **[GitHub repo](https://github.com/tardellirs/mteb-br)** — task definitions, evaluation scripts, paper sources, issue templates |
| - 📄 **[Paper](https://arxiv.org/abs/2607.04581)** on arXiv (MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese) |
|
|
| ## Submit a model |
|
|
| We accept submissions via either channel — pick whichever fits: |
|
|
| - 💬 [HF Discussion on the results dataset](https://huggingface.co/datasets/MTEB-BR/mteb-pt-results/discussions/new) |
| - 🐛 [GitHub Issue with the model template](https://github.com/tardellirs/mteb-br/issues/new?template=submit-model.yml) |
|
|
| Required for a submission: |
| 1. `model_id` (HF repo path or vendor product name) |
| 2. Per-task result JSONs for the 16 headline tasks |
| 3. Reproducible evaluation command |
|
|
| We re-run a sample of each submission to verify before merging. |
|
|
| ## Propose a new task |
|
|
| Open a [GitHub Issue with the task template](https://github.com/tardellirs/mteb-br/issues/new?template=propose-task.yml) describing the dataset, license, size, and discrimination evidence. A task is accepted if it's native PT-BR (not machine-translated), has clear licensing, and discriminates between models. |
|
|
| ## Maintainer |
|
|
| **Tardelli Stekel** — IFSP, São Paulo, Brazil |
| ✉️ <stekel@ifsp.edu.br> |
|
|
| Contributions, corrections, and discussion all welcome. |
|
|
| ## Citation |
|
|
| ```bibtex |
| @misc{mteb-br-2026, |
| title = {MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese}, |
| author = {Stekel, Tardelli R. C.}, |
| year = {2026}, |
| eprint = {2607.04581}, |
| archivePrefix = {arXiv}, |
| primaryClass = {cs.CL}, |
| doi = {10.48550/arXiv.2607.04581}, |
| url = {https://arxiv.org/abs/2607.04581} |
| } |
| ``` |
|
|
| ## Acknowledgments |
|
|
| Built on top of the [`mteb`](https://github.com/embeddings-benchmark/mteb) library by Enevoldsen et al. (2025). Task datasets contributed by their original authors. Compute provided by Modal. |
|
|
