Sentence Similarity
sentence-transformers
ONNX
Safetensors
OpenVINO
multilingual
modernbert
embeddings
feature-extraction
matryoshka
retrieval
text-embeddings-inference
Instructions to use hotchpotch/bekko-embedding-v1-a25m with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- sentence-transformers
How to use hotchpotch/bekko-embedding-v1-a25m with sentence-transformers:
from sentence_transformers import SentenceTransformer model = SentenceTransformer("hotchpotch/bekko-embedding-v1-a25m") sentences = [ "The weather is lovely today.", "It's so sunny outside!", "He drove to the stadium." ] embeddings = model.encode(sentences) similarities = model.similarity(embeddings, embeddings) print(similarities.shape) # [3, 3] - Inference
- Notebooks
- Google Colab
- Kaggle
Translate Japanese snippets and simplify prose punctuation
Browse files
README.md
CHANGED
|
@@ -79,7 +79,7 @@ model = SentenceTransformer(
|
|
| 79 |
query = "What are the characteristics of sushi?"
|
| 80 |
docs = [
|
| 81 |
"A warm noodle soup served in broth with sliced toppings.",
|
| 82 |
-
"倩γ·γγ―ιγιθγ«θ‘£γγ€γγ¦ζγγζηγ§γγ",
|
| 83 |
"Une fine crepe garnie de sucre, de beurre ou de fruits.",
|
| 84 |
"A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
|
| 85 |
]
|
|
@@ -111,7 +111,10 @@ corpus = [
|
|
| 111 |
]
|
| 112 |
corpus_emb = model.encode(corpus, normalize_embeddings=True)
|
| 113 |
|
| 114 |
-
for query in [
|
|
|
|
|
|
|
|
|
|
| 115 |
query_emb = model.encode(query, normalize_embeddings=True)
|
| 116 |
hits = util.semantic_search(query_emb, corpus_emb, top_k=2)[0]
|
| 117 |
print(query)
|
|
@@ -143,7 +146,7 @@ On the 131-task MMTEB Multilingual v2 suite, a25m scores 57.5 Retrieval and 58.3
|
|
| 143 |
<details>
|
| 144 |
<summary>MMTEB Multilingual v2 comparison (131 tasks)</summary>
|
| 145 |
|
| 146 |
-
Scores are Γ100. Retrieval is task-macro nDCG@10
|
| 147 |
|
| 148 |
| Model | Active Params | Dims | Mean | Retrieval | Reranking | BitextMining | STS |
|
| 149 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
@@ -162,7 +165,7 @@ Scores are Γ100. Retrieval is task-macro nDCG@10; Mean is the mean across all 1
|
|
| 162 |
<details>
|
| 163 |
<summary>Full MMTEB Retrieval: all 18 tasks and representative models</summary>
|
| 164 |
|
| 165 |
-
Scores are Γ100. a25m is stronger than a8m on 13 of 18 tasks and on the mean
|
| 166 |
|
| 167 |
| Task | a8m | a25m | mE5-s | G97 | GTE | BGE-M3 |
|
| 168 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
@@ -186,7 +189,7 @@ Scores are Γ100. a25m is stronger than a8m on 13 of 18 tasks and on the mean; i
|
|
| 186 |
| COVID | 72.01 | 73.69 | 72.82 | 70.10 | 80.61 | 77.51 |
|
| 187 |
| MIRACL HN | 53.50 | 59.77 | 60.09 | 56.09 | 64.17 | 69.59 |
|
| 188 |
|
| 189 |
-
Abbreviations: mE5-s = multilingual-e5-small
|
| 190 |
|
| 191 |
</details>
|
| 192 |
|
|
@@ -221,7 +224,7 @@ What each column means:
|
|
| 221 |
<details>
|
| 222 |
<summary>NanoMMTEB-v2: all 18 tasks and representative models</summary>
|
| 223 |
|
| 224 |
-
Scores are nDCG@10. a25m scores higher than a8m on 14 of 18 tasks
|
| 225 |
|
| 226 |
| Task | a8m | a25m | mE5-s | G97 | GTE | BGE-M3 |
|
| 227 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
@@ -292,15 +295,15 @@ Document throughput uses Natural Questions text, batch size 64 and max length 51
|
|
| 292 |
| mE5-large | 303.9M | 21 | 1.5 | 67 | 1,318 |
|
| 293 |
| BGE-M3 | 311.8M | β | β | 78 | 1,324 |
|
| 294 |
|
| 295 |
-
Abbreviations: mE5 = multilingual-e5
|
| 296 |
|
| 297 |
-
Throughput depends on input lengths, batch size, runtime, and hardware. OpenVINO is recommended for CPU
|
| 298 |
|
| 299 |
</details>
|
| 300 |
|
| 301 |
## Optimized Inference
|
| 302 |
|
| 303 |
-
The short version: on CPU,
|
| 304 |
|
| 305 |
<details>
|
| 306 |
<summary>NVIDIA GPU</summary>
|
|
@@ -323,7 +326,7 @@ model = SentenceTransformer(
|
|
| 323 |
query = "What are the characteristics of sushi?"
|
| 324 |
docs = [
|
| 325 |
"A warm noodle soup served in broth with sliced toppings.",
|
| 326 |
-
"倩γ·γγ―ιγιθγ«θ‘£γγ€γγ¦ζγγζηγ§γγ",
|
| 327 |
"Une fine crepe garnie de sucre, de beurre ou de fruits.",
|
| 328 |
"A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
|
| 329 |
]
|
|
@@ -354,7 +357,7 @@ model = SentenceTransformer(
|
|
| 354 |
query = "What are the characteristics of sushi?"
|
| 355 |
docs = [
|
| 356 |
"A warm noodle soup served in broth with sliced toppings.",
|
| 357 |
-
"倩γ·γγ―ιγιθγ«θ‘£γγ€γγ¦ζγγζηγ§γγ",
|
| 358 |
"Une fine crepe garnie de sucre, de beurre ou de fruits.",
|
| 359 |
"A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
|
| 360 |
]
|
|
@@ -447,7 +450,7 @@ const embedding = await extractor("What are the characteristics of sushi?", {
|
|
| 447 |
|
| 448 |
For a complete client-side example, see the [bekko-embedding-web](https://huggingface.co/spaces/hotchpotch/bekko-embedding-web) Space.
|
| 449 |
|
| 450 |
-
`dtype: "fp32"` selects `onnx/model.onnx`
|
| 451 |
|
| 452 |
</details>
|
| 453 |
|
|
@@ -469,7 +472,7 @@ model = SentenceTransformer(
|
|
| 469 |
query = "What are the characteristics of sushi?"
|
| 470 |
docs = [
|
| 471 |
"A warm noodle soup served in broth with sliced toppings.",
|
| 472 |
-
"倩γ·γγ―ιγιθγ«θ‘£γγ€γγ¦ζγγζηγ§γγ",
|
| 473 |
"Une fine crepe garnie de sucre, de beurre ou de fruits.",
|
| 474 |
"A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
|
| 475 |
]
|
|
@@ -508,7 +511,7 @@ import numpy as np
|
|
| 508 |
query = "What are the characteristics of sushi?"
|
| 509 |
docs = [
|
| 510 |
"A warm noodle soup served in broth with sliced toppings.",
|
| 511 |
-
"倩γ·γγ―ιγιθγ«θ‘£γγ€γγ¦ζγγζηγ§γγ",
|
| 512 |
"Une fine crepe garnie de sucre, de beurre ou de fruits.",
|
| 513 |
"A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
|
| 514 |
]
|
|
@@ -547,7 +550,7 @@ def embed(texts):
|
|
| 547 |
query = "What are the characteristics of sushi?"
|
| 548 |
docs = [
|
| 549 |
"A warm noodle soup served in broth with sliced toppings.",
|
| 550 |
-
"倩γ·γγ―ιγιθγ«θ‘£γγ€γγ¦ζγγζηγ§γγ",
|
| 551 |
"Une fine crepe garnie de sucre, de beurre ou de fruits.",
|
| 552 |
"A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
|
| 553 |
]
|
|
@@ -580,10 +583,10 @@ How much quality do you trade for a smaller index? For `bekko-embedding-v1-a25m`
|
|
| 580 |
|
| 581 |
## FAQ
|
| 582 |
|
| 583 |
-
- *Do I need a prefix like `query: ` or `passage: `?* β No. bekko is trained without prefixes
|
| 584 |
- *Which languages are covered?* β 100+ languages, inherited from the mmBERT base model. Coverage is broad but uneven, so evaluate on your own language and domain before deployment (see [Limitations](#limitations)).
|
| 585 |
- *Which file should I load for my runtime?* β PyTorch: the default safetensors weights. Fastest CPU inference: `openvino/openvino_model.xml`. Browser / ONNX Runtime: `onnx/model.onnx`. Files named `_not_default` / `_not_recommended` are comparison artifacts, not deployment choices.
|
| 586 |
-
- *Can I make the embeddings smaller?* β Yes β pass `truncate_dim=256` (or 128 / 64)
|
| 587 |
- *Can it really run in a browser?* β Yes. Try the [bekko-embedding-web](https://huggingface.co/spaces/hotchpotch/bekko-embedding-web) demo β the model runs fully client-side with Transformers.js.
|
| 588 |
|
| 589 |
## Limitations
|
|
@@ -591,8 +594,8 @@ How much quality do you trade for a smaller index? For `bekko-embedding-v1-a25m`
|
|
| 591 |
<details>
|
| 592 |
<summary>Evaluation scope and deployment considerations</summary>
|
| 593 |
|
| 594 |
-
- Bekko is optimized primarily for multilingual retrieval. Its strongest MMTEB results are Retrieval, Reranking, BitextMining, and STS
|
| 595 |
-
- Bekko is a bi-encoder embedding model, not a cross-encoder reranker. MMTEB Reranking scores measure bi-encoder similarity scoring
|
| 596 |
- Support for 100+ languages reflects training-data coverage. Quality varies by language and domain, so evaluate on your target data before deployment.
|
| 597 |
- HAKARI-Bench is maintained by the model author and should be read alongside the independently maintained MMTEB suite. Bekko's MMTEB results use the same 131-task set and aggregation rules as the referenced snapshot, but await submission through the official leaderboard pipeline.
|
| 598 |
- Throughput varies with text lengths, batch size, backend, software versions, and hardware. Use the benchmark figures as comparative measurements, not guaranteed production latency.
|
|
|
|
| 79 |
query = "What are the characteristics of sushi?"
|
| 80 |
docs = [
|
| 81 |
"A warm noodle soup served in broth with sliced toppings.",
|
| 82 |
+
"倩γ·γγ―ιγιθγ«θ‘£γγ€γγ¦ζγγζηγ§γγ", # "Tempura is battered, deep-fried fish and vegetables."
|
| 83 |
"Une fine crepe garnie de sucre, de beurre ou de fruits.",
|
| 84 |
"A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
|
| 85 |
]
|
|
|
|
| 111 |
]
|
| 112 |
corpus_emb = model.encode(corpus, normalize_embeddings=True)
|
| 113 |
|
| 114 |
+
for query in [
|
| 115 |
+
"ζ₯ζ¬γ§δΈηͺι«γε±±γ―οΌ", # "What is the highest mountain in Japan?"
|
| 116 |
+
"Who designed the famous basilica in Barcelona?",
|
| 117 |
+
]:
|
| 118 |
query_emb = model.encode(query, normalize_embeddings=True)
|
| 119 |
hits = util.semantic_search(query_emb, corpus_emb, top_k=2)[0]
|
| 120 |
print(query)
|
|
|
|
| 146 |
<details>
|
| 147 |
<summary>MMTEB Multilingual v2 comparison (131 tasks)</summary>
|
| 148 |
|
| 149 |
+
Scores are Γ100. Retrieval is task-macro nDCG@10, and Mean is the mean across all 131 tasks. Competitor values use the official 2026-06-28 snapshot. Bekko was evaluated over the same task set and aggregation rules.
|
| 150 |
|
| 151 |
| Model | Active Params | Dims | Mean | Retrieval | Reranking | BitextMining | STS |
|
| 152 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
|
|
| 165 |
<details>
|
| 166 |
<summary>Full MMTEB Retrieval: all 18 tasks and representative models</summary>
|
| 167 |
|
| 168 |
+
Scores are Γ100. a25m is stronger than a8m on 13 of 18 tasks and on the mean. Its main regression is WinoGrande.
|
| 169 |
|
| 170 |
| Task | a8m | a25m | mE5-s | G97 | GTE | BGE-M3 |
|
| 171 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
|
|
| 189 |
| COVID | 72.01 | 73.69 | 72.82 | 70.10 | 80.61 | 77.51 |
|
| 190 |
| MIRACL HN | 53.50 | 59.77 | 60.09 | 56.09 | 64.17 | 69.59 |
|
| 191 |
|
| 192 |
+
Abbreviations: mE5-s = multilingual-e5-small, G97 = Granite Embedding 97M Multilingual R2, GTE = gte-multilingual-base, MIRACL HN = MIRACL Retrieval Hard Negatives.
|
| 193 |
|
| 194 |
</details>
|
| 195 |
|
|
|
|
| 224 |
<details>
|
| 225 |
<summary>NanoMMTEB-v2: all 18 tasks and representative models</summary>
|
| 226 |
|
| 227 |
+
Scores are nDCG@10. a25m scores higher than a8m on 14 of 18 tasks. Its slightly lower simple mean is mainly due to LEMBPasskey.
|
| 228 |
|
| 229 |
| Task | a8m | a25m | mE5-s | G97 | GTE | BGE-M3 |
|
| 230 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
|
|
| 295 |
| mE5-large | 303.9M | 21 | 1.5 | 67 | 1,318 |
|
| 296 |
| BGE-M3 | 311.8M | β | β | 78 | 1,324 |
|
| 297 |
|
| 298 |
+
Abbreviations: mE5 = multilingual-e5, granite-97m/311m-r2 = Granite Embedding Multilingual R2, EmbGemma = EmbeddingGemma. x86 = Ryzen 9 7950X + OpenVINO, Pi 5 = Raspberry Pi 5 + OpenVINO, M4 = Apple M4 Max + MPS, RTX = RTX 5090 + CUDA/Flash Attention 2. AP means active parameters.
|
| 299 |
|
| 300 |
+
Throughput depends on input lengths, batch size, runtime, and hardware. OpenVINO is recommended for CPU, MPS for Apple Silicon, and Flash Attention 2 for supported NVIDIA GPUs.
|
| 301 |
|
| 302 |
</details>
|
| 303 |
|
| 304 |
## Optimized Inference
|
| 305 |
|
| 306 |
+
The short version: use the default OpenVINO artifact on CPU, and the default ONNX artifact in the browser or with ONNX Runtime. Both keep the tokenizer and vocabulary untouched and compress only the static token embedding table.
|
| 307 |
|
| 308 |
<details>
|
| 309 |
<summary>NVIDIA GPU</summary>
|
|
|
|
| 326 |
query = "What are the characteristics of sushi?"
|
| 327 |
docs = [
|
| 328 |
"A warm noodle soup served in broth with sliced toppings.",
|
| 329 |
+
"倩γ·γγ―ιγιθγ«θ‘£γγ€γγ¦ζγγζηγ§γγ", # "Tempura is battered, deep-fried fish and vegetables."
|
| 330 |
"Une fine crepe garnie de sucre, de beurre ou de fruits.",
|
| 331 |
"A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
|
| 332 |
]
|
|
|
|
| 357 |
query = "What are the characteristics of sushi?"
|
| 358 |
docs = [
|
| 359 |
"A warm noodle soup served in broth with sliced toppings.",
|
| 360 |
+
"倩γ·γγ―ιγιθγ«θ‘£γγ€γγ¦ζγγζηγ§γγ", # "Tempura is battered, deep-fried fish and vegetables."
|
| 361 |
"Une fine crepe garnie de sucre, de beurre ou de fruits.",
|
| 362 |
"A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
|
| 363 |
]
|
|
|
|
| 450 |
|
| 451 |
For a complete client-side example, see the [bekko-embedding-web](https://huggingface.co/spaces/hotchpotch/bekko-embedding-web) Space.
|
| 452 |
|
| 453 |
+
`dtype: "fp32"` selects `onnx/model.onnx`. In this repository, that filename is the compact default (static embedding table in `int8`), while Transformer computation remains `fp32`. Additional `fp16`, `fp32`, and ONNX `qint8`/`quint8` comparison files are included under explicit names. The Transformer-weight quantized files are experimental, not default choices.
|
| 454 |
|
| 455 |
</details>
|
| 456 |
|
|
|
|
| 472 |
query = "What are the characteristics of sushi?"
|
| 473 |
docs = [
|
| 474 |
"A warm noodle soup served in broth with sliced toppings.",
|
| 475 |
+
"倩γ·γγ―ιγιθγ«θ‘£γγ€γγ¦ζγγζηγ§γγ", # "Tempura is battered, deep-fried fish and vegetables."
|
| 476 |
"Une fine crepe garnie de sucre, de beurre ou de fruits.",
|
| 477 |
"A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
|
| 478 |
]
|
|
|
|
| 511 |
query = "What are the characteristics of sushi?"
|
| 512 |
docs = [
|
| 513 |
"A warm noodle soup served in broth with sliced toppings.",
|
| 514 |
+
"倩γ·γγ―ιγιθγ«θ‘£γγ€γγ¦ζγγζηγ§γγ", # "Tempura is battered, deep-fried fish and vegetables."
|
| 515 |
"Une fine crepe garnie de sucre, de beurre ou de fruits.",
|
| 516 |
"A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
|
| 517 |
]
|
|
|
|
| 550 |
query = "What are the characteristics of sushi?"
|
| 551 |
docs = [
|
| 552 |
"A warm noodle soup served in broth with sliced toppings.",
|
| 553 |
+
"倩γ·γγ―ιγιθγ«θ‘£γγ€γγ¦ζγγζηγ§γγ", # "Tempura is battered, deep-fried fish and vegetables."
|
| 554 |
"Une fine crepe garnie de sucre, de beurre ou de fruits.",
|
| 555 |
"A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
|
| 556 |
]
|
|
|
|
| 583 |
|
| 584 |
## FAQ
|
| 585 |
|
| 586 |
+
- *Do I need a prefix like `query: ` or `passage: `?* β No. bekko is trained without prefixes, so you encode raw text for both queries and documents. If you come from the multilingual-e5 family, just drop the prefixes.
|
| 587 |
- *Which languages are covered?* β 100+ languages, inherited from the mmBERT base model. Coverage is broad but uneven, so evaluate on your own language and domain before deployment (see [Limitations](#limitations)).
|
| 588 |
- *Which file should I load for my runtime?* β PyTorch: the default safetensors weights. Fastest CPU inference: `openvino/openvino_model.xml`. Browser / ONNX Runtime: `onnx/model.onnx`. Files named `_not_default` / `_not_recommended` are comparison artifacts, not deployment choices.
|
| 589 |
+
- *Can I make the embeddings smaller?* β Yes β pass `truncate_dim=256` (or 128 / 64). See [Truncation and Quantization](#truncation-and-quantization) for the quality cost.
|
| 590 |
- *Can it really run in a browser?* β Yes. Try the [bekko-embedding-web](https://huggingface.co/spaces/hotchpotch/bekko-embedding-web) demo β the model runs fully client-side with Transformers.js.
|
| 591 |
|
| 592 |
## Limitations
|
|
|
|
| 594 |
<details>
|
| 595 |
<summary>Evaluation scope and deployment considerations</summary>
|
| 596 |
|
| 597 |
+
- Bekko is optimized primarily for multilingual retrieval. Its strongest MMTEB results are Retrieval, Reranking, BitextMining, and STS. It is not intended to be state of the art across every embedding task category.
|
| 598 |
+
- Bekko is a bi-encoder embedding model, not a cross-encoder reranker. MMTEB Reranking scores measure bi-encoder similarity scoring. Use a dedicated cross-encoder when maximum reranking accuracy is more important than throughput.
|
| 599 |
- Support for 100+ languages reflects training-data coverage. Quality varies by language and domain, so evaluate on your target data before deployment.
|
| 600 |
- HAKARI-Bench is maintained by the model author and should be read alongside the independently maintained MMTEB suite. Bekko's MMTEB results use the same 131-task set and aggregation rules as the referenced snapshot, but await submission through the official leaderboard pipeline.
|
| 601 |
- Throughput varies with text lengths, batch size, backend, software versions, and hardware. Use the benchmark figures as comparative measurements, not guaranteed production latency.
|