hotchpotch commited on
Commit
8573fd4
Β·
verified Β·
1 Parent(s): 25aaa4d

Translate Japanese snippets and simplify prose punctuation

Browse files
Files changed (1) hide show
  1. README.md +22 -19
README.md CHANGED
@@ -79,7 +79,7 @@ model = SentenceTransformer(
79
  query = "What are the characteristics of sushi?"
80
  docs = [
81
  "A warm noodle soup served in broth with sliced toppings.",
82
- "ε€©γ·γ‚‰γ―ι­šγ‚„ι‡Žθœγ«θ‘£γ‚’γ€γ‘γ¦ζšγ’γŸζ–™η†γ§γ™γ€‚",
83
  "Une fine crepe garnie de sucre, de beurre ou de fruits.",
84
  "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
85
  ]
@@ -111,7 +111,10 @@ corpus = [
111
  ]
112
  corpus_emb = model.encode(corpus, normalize_embeddings=True)
113
 
114
- for query in ["ζ—₯ζœ¬γ§δΈ€η•ͺι«˜γ„ε±±γ―οΌŸ", "Who designed the famous basilica in Barcelona?"]:
 
 
 
115
  query_emb = model.encode(query, normalize_embeddings=True)
116
  hits = util.semantic_search(query_emb, corpus_emb, top_k=2)[0]
117
  print(query)
@@ -143,7 +146,7 @@ On the 131-task MMTEB Multilingual v2 suite, a25m scores 57.5 Retrieval and 58.3
143
  <details>
144
  <summary>MMTEB Multilingual v2 comparison (131 tasks)</summary>
145
 
146
- Scores are Γ—100. Retrieval is task-macro nDCG@10; Mean is the mean across all 131 tasks. Competitor values use the official 2026-06-28 snapshot; Bekko was evaluated over the same task set and aggregation rules.
147
 
148
  | Model | Active Params | Dims | Mean | Retrieval | Reranking | BitextMining | STS |
149
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
@@ -162,7 +165,7 @@ Scores are Γ—100. Retrieval is task-macro nDCG@10; Mean is the mean across all 1
162
  <details>
163
  <summary>Full MMTEB Retrieval: all 18 tasks and representative models</summary>
164
 
165
- Scores are Γ—100. a25m is stronger than a8m on 13 of 18 tasks and on the mean; its main regression is WinoGrande.
166
 
167
  | Task | a8m | a25m | mE5-s | G97 | GTE | BGE-M3 |
168
  | --- | ---: | ---: | ---: | ---: | ---: | ---: |
@@ -186,7 +189,7 @@ Scores are Γ—100. a25m is stronger than a8m on 13 of 18 tasks and on the mean; i
186
  | COVID | 72.01 | 73.69 | 72.82 | 70.10 | 80.61 | 77.51 |
187
  | MIRACL HN | 53.50 | 59.77 | 60.09 | 56.09 | 64.17 | 69.59 |
188
 
189
- Abbreviations: mE5-s = multilingual-e5-small; G97 = Granite Embedding 97M Multilingual R2; GTE = gte-multilingual-base; MIRACL HN = MIRACL Retrieval Hard Negatives.
190
 
191
  </details>
192
 
@@ -221,7 +224,7 @@ What each column means:
221
  <details>
222
  <summary>NanoMMTEB-v2: all 18 tasks and representative models</summary>
223
 
224
- Scores are nDCG@10. a25m scores higher than a8m on 14 of 18 tasks; its slightly lower simple mean is mainly due to LEMBPasskey.
225
 
226
  | Task | a8m | a25m | mE5-s | G97 | GTE | BGE-M3 |
227
  | --- | ---: | ---: | ---: | ---: | ---: | ---: |
@@ -292,15 +295,15 @@ Document throughput uses Natural Questions text, batch size 64 and max length 51
292
  | mE5-large | 303.9M | 21 | 1.5 | 67 | 1,318 |
293
  | BGE-M3 | 311.8M | β€” | β€” | 78 | 1,324 |
294
 
295
- Abbreviations: mE5 = multilingual-e5; granite-97m/311m-r2 = Granite Embedding Multilingual R2; EmbGemma = EmbeddingGemma. x86 = Ryzen 9 7950X + OpenVINO; Pi 5 = Raspberry Pi 5 + OpenVINO; M4 = Apple M4 Max + MPS; RTX = RTX 5090 + CUDA/Flash Attention 2. AP means active parameters.
296
 
297
- Throughput depends on input lengths, batch size, runtime, and hardware. OpenVINO is recommended for CPU; MPS for Apple Silicon; Flash Attention 2 for supported NVIDIA GPUs.
298
 
299
  </details>
300
 
301
  ## Optimized Inference
302
 
303
- The short version: on CPU, use the default OpenVINO artifact; in the browser or with ONNX Runtime, use the default ONNX artifact. Both keep the tokenizer and vocabulary untouched and compress only the static token embedding table.
304
 
305
  <details>
306
  <summary>NVIDIA GPU</summary>
@@ -323,7 +326,7 @@ model = SentenceTransformer(
323
  query = "What are the characteristics of sushi?"
324
  docs = [
325
  "A warm noodle soup served in broth with sliced toppings.",
326
- "ε€©γ·γ‚‰γ―ι­šγ‚„ι‡Žθœγ«θ‘£γ‚’γ€γ‘γ¦ζšγ’γŸζ–™η†γ§γ™γ€‚",
327
  "Une fine crepe garnie de sucre, de beurre ou de fruits.",
328
  "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
329
  ]
@@ -354,7 +357,7 @@ model = SentenceTransformer(
354
  query = "What are the characteristics of sushi?"
355
  docs = [
356
  "A warm noodle soup served in broth with sliced toppings.",
357
- "ε€©γ·γ‚‰γ―ι­šγ‚„ι‡Žθœγ«θ‘£γ‚’γ€γ‘γ¦ζšγ’γŸζ–™η†γ§γ™γ€‚",
358
  "Une fine crepe garnie de sucre, de beurre ou de fruits.",
359
  "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
360
  ]
@@ -447,7 +450,7 @@ const embedding = await extractor("What are the characteristics of sushi?", {
447
 
448
  For a complete client-side example, see the [bekko-embedding-web](https://huggingface.co/spaces/hotchpotch/bekko-embedding-web) Space.
449
 
450
- `dtype: "fp32"` selects `onnx/model.onnx`; in this repository that filename is the compact default (static embedding table in `int8`), while Transformer computation remains `fp32`. Additional `fp16`, `fp32`, and ONNX `qint8`/`quint8` comparison files are included under explicit names. The Transformer-weight quantized files are experimental, not default choices.
451
 
452
  </details>
453
 
@@ -469,7 +472,7 @@ model = SentenceTransformer(
469
  query = "What are the characteristics of sushi?"
470
  docs = [
471
  "A warm noodle soup served in broth with sliced toppings.",
472
- "ε€©γ·γ‚‰γ―ι­šγ‚„ι‡Žθœγ«θ‘£γ‚’γ€γ‘γ¦ζšγ’γŸζ–™η†γ§γ™γ€‚",
473
  "Une fine crepe garnie de sucre, de beurre ou de fruits.",
474
  "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
475
  ]
@@ -508,7 +511,7 @@ import numpy as np
508
  query = "What are the characteristics of sushi?"
509
  docs = [
510
  "A warm noodle soup served in broth with sliced toppings.",
511
- "ε€©γ·γ‚‰γ―ι­šγ‚„ι‡Žθœγ«θ‘£γ‚’γ€γ‘γ¦ζšγ’γŸζ–™η†γ§γ™γ€‚",
512
  "Une fine crepe garnie de sucre, de beurre ou de fruits.",
513
  "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
514
  ]
@@ -547,7 +550,7 @@ def embed(texts):
547
  query = "What are the characteristics of sushi?"
548
  docs = [
549
  "A warm noodle soup served in broth with sliced toppings.",
550
- "ε€©γ·γ‚‰γ―ι­šγ‚„ι‡Žθœγ«θ‘£γ‚’γ€γ‘γ¦ζšγ’γŸζ–™η†γ§γ™γ€‚",
551
  "Une fine crepe garnie de sucre, de beurre ou de fruits.",
552
  "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
553
  ]
@@ -580,10 +583,10 @@ How much quality do you trade for a smaller index? For `bekko-embedding-v1-a25m`
580
 
581
  ## FAQ
582
 
583
- - *Do I need a prefix like `query: ` or `passage: `?* β€” No. bekko is trained without prefixes; encode raw text for both queries and documents. If you're migrating from the multilingual-e5 family, simply drop the prefixes.
584
  - *Which languages are covered?* β€” 100+ languages, inherited from the mmBERT base model. Coverage is broad but uneven, so evaluate on your own language and domain before deployment (see [Limitations](#limitations)).
585
  - *Which file should I load for my runtime?* β€” PyTorch: the default safetensors weights. Fastest CPU inference: `openvino/openvino_model.xml`. Browser / ONNX Runtime: `onnx/model.onnx`. Files named `_not_default` / `_not_recommended` are comparison artifacts, not deployment choices.
586
- - *Can I make the embeddings smaller?* β€” Yes β€” pass `truncate_dim=256` (or 128 / 64); see [Truncation and Quantization](#truncation-and-quantization) for the quality cost.
587
  - *Can it really run in a browser?* β€” Yes. Try the [bekko-embedding-web](https://huggingface.co/spaces/hotchpotch/bekko-embedding-web) demo β€” the model runs fully client-side with Transformers.js.
588
 
589
  ## Limitations
@@ -591,8 +594,8 @@ How much quality do you trade for a smaller index? For `bekko-embedding-v1-a25m`
591
  <details>
592
  <summary>Evaluation scope and deployment considerations</summary>
593
 
594
- - Bekko is optimized primarily for multilingual retrieval. Its strongest MMTEB results are Retrieval, Reranking, BitextMining, and STS; it is not intended to be state of the art across every embedding task category.
595
- - Bekko is a bi-encoder embedding model, not a cross-encoder reranker. MMTEB Reranking scores measure bi-encoder similarity scoring; use a dedicated cross-encoder when maximum reranking accuracy is more important than throughput.
596
  - Support for 100+ languages reflects training-data coverage. Quality varies by language and domain, so evaluate on your target data before deployment.
597
  - HAKARI-Bench is maintained by the model author and should be read alongside the independently maintained MMTEB suite. Bekko's MMTEB results use the same 131-task set and aggregation rules as the referenced snapshot, but await submission through the official leaderboard pipeline.
598
  - Throughput varies with text lengths, batch size, backend, software versions, and hardware. Use the benchmark figures as comparative measurements, not guaranteed production latency.
 
79
  query = "What are the characteristics of sushi?"
80
  docs = [
81
  "A warm noodle soup served in broth with sliced toppings.",
82
+ "ε€©γ·γ‚‰γ―ι­šγ‚„ι‡Žθœγ«θ‘£γ‚’γ€γ‘γ¦ζšγ’γŸζ–™η†γ§γ™γ€‚", # "Tempura is battered, deep-fried fish and vegetables."
83
  "Une fine crepe garnie de sucre, de beurre ou de fruits.",
84
  "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
85
  ]
 
111
  ]
112
  corpus_emb = model.encode(corpus, normalize_embeddings=True)
113
 
114
+ for query in [
115
+ "ζ—₯ζœ¬γ§δΈ€η•ͺι«˜γ„ε±±γ―οΌŸ", # "What is the highest mountain in Japan?"
116
+ "Who designed the famous basilica in Barcelona?",
117
+ ]:
118
  query_emb = model.encode(query, normalize_embeddings=True)
119
  hits = util.semantic_search(query_emb, corpus_emb, top_k=2)[0]
120
  print(query)
 
146
  <details>
147
  <summary>MMTEB Multilingual v2 comparison (131 tasks)</summary>
148
 
149
+ Scores are Γ—100. Retrieval is task-macro nDCG@10, and Mean is the mean across all 131 tasks. Competitor values use the official 2026-06-28 snapshot. Bekko was evaluated over the same task set and aggregation rules.
150
 
151
  | Model | Active Params | Dims | Mean | Retrieval | Reranking | BitextMining | STS |
152
  | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
 
165
  <details>
166
  <summary>Full MMTEB Retrieval: all 18 tasks and representative models</summary>
167
 
168
+ Scores are Γ—100. a25m is stronger than a8m on 13 of 18 tasks and on the mean. Its main regression is WinoGrande.
169
 
170
  | Task | a8m | a25m | mE5-s | G97 | GTE | BGE-M3 |
171
  | --- | ---: | ---: | ---: | ---: | ---: | ---: |
 
189
  | COVID | 72.01 | 73.69 | 72.82 | 70.10 | 80.61 | 77.51 |
190
  | MIRACL HN | 53.50 | 59.77 | 60.09 | 56.09 | 64.17 | 69.59 |
191
 
192
+ Abbreviations: mE5-s = multilingual-e5-small, G97 = Granite Embedding 97M Multilingual R2, GTE = gte-multilingual-base, MIRACL HN = MIRACL Retrieval Hard Negatives.
193
 
194
  </details>
195
 
 
224
  <details>
225
  <summary>NanoMMTEB-v2: all 18 tasks and representative models</summary>
226
 
227
+ Scores are nDCG@10. a25m scores higher than a8m on 14 of 18 tasks. Its slightly lower simple mean is mainly due to LEMBPasskey.
228
 
229
  | Task | a8m | a25m | mE5-s | G97 | GTE | BGE-M3 |
230
  | --- | ---: | ---: | ---: | ---: | ---: | ---: |
 
295
  | mE5-large | 303.9M | 21 | 1.5 | 67 | 1,318 |
296
  | BGE-M3 | 311.8M | β€” | β€” | 78 | 1,324 |
297
 
298
+ Abbreviations: mE5 = multilingual-e5, granite-97m/311m-r2 = Granite Embedding Multilingual R2, EmbGemma = EmbeddingGemma. x86 = Ryzen 9 7950X + OpenVINO, Pi 5 = Raspberry Pi 5 + OpenVINO, M4 = Apple M4 Max + MPS, RTX = RTX 5090 + CUDA/Flash Attention 2. AP means active parameters.
299
 
300
+ Throughput depends on input lengths, batch size, runtime, and hardware. OpenVINO is recommended for CPU, MPS for Apple Silicon, and Flash Attention 2 for supported NVIDIA GPUs.
301
 
302
  </details>
303
 
304
  ## Optimized Inference
305
 
306
+ The short version: use the default OpenVINO artifact on CPU, and the default ONNX artifact in the browser or with ONNX Runtime. Both keep the tokenizer and vocabulary untouched and compress only the static token embedding table.
307
 
308
  <details>
309
  <summary>NVIDIA GPU</summary>
 
326
  query = "What are the characteristics of sushi?"
327
  docs = [
328
  "A warm noodle soup served in broth with sliced toppings.",
329
+ "ε€©γ·γ‚‰γ―ι­šγ‚„ι‡Žθœγ«θ‘£γ‚’γ€γ‘γ¦ζšγ’γŸζ–™η†γ§γ™γ€‚", # "Tempura is battered, deep-fried fish and vegetables."
330
  "Une fine crepe garnie de sucre, de beurre ou de fruits.",
331
  "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
332
  ]
 
357
  query = "What are the characteristics of sushi?"
358
  docs = [
359
  "A warm noodle soup served in broth with sliced toppings.",
360
+ "ε€©γ·γ‚‰γ―ι­šγ‚„ι‡Žθœγ«θ‘£γ‚’γ€γ‘γ¦ζšγ’γŸζ–™η†γ§γ™γ€‚", # "Tempura is battered, deep-fried fish and vegetables."
361
  "Une fine crepe garnie de sucre, de beurre ou de fruits.",
362
  "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
363
  ]
 
450
 
451
  For a complete client-side example, see the [bekko-embedding-web](https://huggingface.co/spaces/hotchpotch/bekko-embedding-web) Space.
452
 
453
+ `dtype: "fp32"` selects `onnx/model.onnx`. In this repository, that filename is the compact default (static embedding table in `int8`), while Transformer computation remains `fp32`. Additional `fp16`, `fp32`, and ONNX `qint8`/`quint8` comparison files are included under explicit names. The Transformer-weight quantized files are experimental, not default choices.
454
 
455
  </details>
456
 
 
472
  query = "What are the characteristics of sushi?"
473
  docs = [
474
  "A warm noodle soup served in broth with sliced toppings.",
475
+ "ε€©γ·γ‚‰γ―ι­šγ‚„ι‡Žθœγ«θ‘£γ‚’γ€γ‘γ¦ζšγ’γŸζ–™η†γ§γ™γ€‚", # "Tempura is battered, deep-fried fish and vegetables."
476
  "Une fine crepe garnie de sucre, de beurre ou de fruits.",
477
  "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
478
  ]
 
511
  query = "What are the characteristics of sushi?"
512
  docs = [
513
  "A warm noodle soup served in broth with sliced toppings.",
514
+ "ε€©γ·γ‚‰γ―ι­šγ‚„ι‡Žθœγ«θ‘£γ‚’γ€γ‘γ¦ζšγ’γŸζ–™η†γ§γ™γ€‚", # "Tempura is battered, deep-fried fish and vegetables."
515
  "Une fine crepe garnie de sucre, de beurre ou de fruits.",
516
  "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
517
  ]
 
550
  query = "What are the characteristics of sushi?"
551
  docs = [
552
  "A warm noodle soup served in broth with sliced toppings.",
553
+ "ε€©γ·γ‚‰γ―ι­šγ‚„ι‡Žθœγ«θ‘£γ‚’γ€γ‘γ¦ζšγ’γŸζ–™η†γ§γ™γ€‚", # "Tempura is battered, deep-fried fish and vegetables."
554
  "Une fine crepe garnie de sucre, de beurre ou de fruits.",
555
  "A Japanese dish made with vinegared rice, often shaped with seafood, vegetables, or egg.",
556
  ]
 
583
 
584
  ## FAQ
585
 
586
+ - *Do I need a prefix like `query: ` or `passage: `?* β€” No. bekko is trained without prefixes, so you encode raw text for both queries and documents. If you come from the multilingual-e5 family, just drop the prefixes.
587
  - *Which languages are covered?* β€” 100+ languages, inherited from the mmBERT base model. Coverage is broad but uneven, so evaluate on your own language and domain before deployment (see [Limitations](#limitations)).
588
  - *Which file should I load for my runtime?* β€” PyTorch: the default safetensors weights. Fastest CPU inference: `openvino/openvino_model.xml`. Browser / ONNX Runtime: `onnx/model.onnx`. Files named `_not_default` / `_not_recommended` are comparison artifacts, not deployment choices.
589
+ - *Can I make the embeddings smaller?* β€” Yes β€” pass `truncate_dim=256` (or 128 / 64). See [Truncation and Quantization](#truncation-and-quantization) for the quality cost.
590
  - *Can it really run in a browser?* β€” Yes. Try the [bekko-embedding-web](https://huggingface.co/spaces/hotchpotch/bekko-embedding-web) demo β€” the model runs fully client-side with Transformers.js.
591
 
592
  ## Limitations
 
594
  <details>
595
  <summary>Evaluation scope and deployment considerations</summary>
596
 
597
+ - Bekko is optimized primarily for multilingual retrieval. Its strongest MMTEB results are Retrieval, Reranking, BitextMining, and STS. It is not intended to be state of the art across every embedding task category.
598
+ - Bekko is a bi-encoder embedding model, not a cross-encoder reranker. MMTEB Reranking scores measure bi-encoder similarity scoring. Use a dedicated cross-encoder when maximum reranking accuracy is more important than throughput.
599
  - Support for 100+ languages reflects training-data coverage. Quality varies by language and domain, so evaluate on your target data before deployment.
600
  - HAKARI-Bench is maintained by the model author and should be read alongside the independently maintained MMTEB suite. Bekko's MMTEB results use the same 131-task set and aggregation rules as the referenced snapshot, but await submission through the official leaderboard pipeline.
601
  - Throughput varies with text lengths, batch size, backend, software versions, and hardware. Use the benchmark figures as comparative measurements, not guaranteed production latency.