Hugging Face's logo

jinaai
/
jina-colbert-v2

sentence-transformers
ONNX
Safetensors
ColBERT
multi-vector
passage-retrieval
custom_code
🇪🇺 Region: EU
Model card Files
xet
 Community
22
nan commited on
Commit
4e69429
·
1 Parent(s): 0eb5b5e

feat: update README

Browse files
Files changed (1) hide show
  1. README.md +35 -34
README.md CHANGED
@@ -213,7 +213,7 @@ print(query_vectors)
213
 
214
  #### BEIR
215
 
216
- | **Short Name** | **jina-colbert-v2** | **jina-colbert-v1** | **ColBERTv2.0** | **BM25** |
217
  |--------------------|---------------------|---------------------|-----------------|----------|
218
  | **avg** | 0.531 | 0.502 | 0.496 | 0.440 |
219
  | **nfcorpus** | 0.346 | 0.338 | 0.337 | 0.325 |
@@ -234,19 +234,16 @@ print(query_vectors)
234
 
235
  #### MS MARCO Passage Retrieval
236
 
237
- | **Models** | **MRR@10** |
238
- |---------------------|------------|
239
- | **jina-colbert-v2** | 0.396 |
240
- | **jina-colbert-v1** | 0.390 |
241
- | **ColBERTv2.0** | **0.397** |
242
- | **BM25** | 0.187 |
243
 
244
  ### Multilingual Benchmarks
245
 
246
  #### MIRACLE
247
- We present our model's performance on the MIRACLE dataset, which is a multilingual retrieval benchmark.
248
 
249
- | **** | **jina-colbert-v2** | **mDPR (zero shot)** |
250
  |---------|---------------------|----------------------|
251
  | **avg** | 0.627 | 0.427 |
252
  | **ar** | 0.753 | 0.499 |
@@ -270,41 +267,45 @@ We present our model's performance on the MIRACLE dataset, which is a multilingu
270
 
271
  #### mMARCO
272
 
273
- | **** | **jina-colbert-v2** | **BM-25** |
274
- |--------|---------------------|-----------|
275
- | **ar** | 0.272 | 0.111 |
276
- | **de** | 0.331 | 0.136 |
277
- | **nl** | 0.330 | 0.140 |
278
- | **es** | 0.341 | 0.158 |
279
- | **fr** | 0.335 | 0.155 |
280
- | **hi** | 0.309 | 0.134 |
281
- | **id** | 0.319 | 0.149 |
282
- | **it** | 0.337 | 0.153 |
283
- | **ja** | 0.276 | 0.141 |
284
- | **pt** | 0.337 | 0.152 |
285
- | **ru** | 0.298 | 0.124 |
286
- | **vi** | 0.287 | 0.136 |
287
- | **zh** | 0.302 | |
 
 
288
 
289
 
290
  ### Matryoshka Representation Benchmarks
291
 
292
  #### BEIR
293
 
294
- | **dim** | **Average** | **nfcorpus** | **fiqa** | **trec-covid** | **hotpotqa** | **nq** |
295
- |---------|-------------|--------------|----------|----------------|--------------|--------|
296
- | **128** | 0.565 | 0.346 | 0.408 | 0.834 | 0.766 | 0.640 |
297
- | **96** | 0.558 | 0.340 | 0.404 | 0.808 | 0.764 | 0.640 |
298
- | **64** | 0.556 | 0.347 | 0.404 | 0.805 | 0.756 | 0.635 |
 
 
 
 
299
 
300
 
301
  #### MSMARCO
302
 
303
- | **dim** | **msmarco** |
304
- |---------|-------------|
305
- | **128** | 0.396 |
306
- | **96** | 0.391 |
307
- | **64** | 0.388 |
308
 
309
  ## Other Models
310
 
 
213
 
214
  #### BEIR
215
 
216
+ | **NDCG@10** | **jina-colbert-v2** | **jina-colbert-v1** | **ColBERTv2.0** | **BM25** |
217
  |--------------------|---------------------|---------------------|-----------------|----------|
218
  | **avg** | 0.531 | 0.502 | 0.496 | 0.440 |
219
  | **nfcorpus** | 0.346 | 0.338 | 0.337 | 0.325 |
 
234
 
235
  #### MS MARCO Passage Retrieval
236
 
237
+ | **MRR@10** | **jina-colbert-v2** | **jina-colbert-v1** | **ColBERTv2.0** | **BM25** |
238
+ |-------------|---------------------|---------------------|-----------------|----------|
239
+ | **MSMARCO** | 0.396 | 0.390 | 0.397 | 0.187 |
240
+
 
 
241
 
242
  ### Multilingual Benchmarks
243
 
244
  #### MIRACLE
 
245
 
246
+ | **NDCG@10** | **jina-colbert-v2** | **mDPR (zero shot)** |
247
  |---------|---------------------|----------------------|
248
  | **avg** | 0.627 | 0.427 |
249
  | **ar** | 0.753 | 0.499 |
 
267
 
268
  #### mMARCO
269
 
270
+ | **MRR@10** | **jina-colbert-v2** | **BM-25** | **ColBERT-XM** |
271
+ |------------|---------------------|-----------|----------------|
272
+ | **avg** | 0.313 | 0.141 | 0.254 |
273
+ | **ar** | 0.272 | 0.111 | 0.195 |
274
+ | **de** | 0.331 | 0.136 | 0.270 |
275
+ | **nl** | 0.330 | 0.140 | 0.275 |
276
+ | **es** | 0.341 | 0.158 | 0.285 |
277
+ | **fr** | 0.335 | 0.155 | 0.269 |
278
+ | **hi** | 0.309 | 0.134 | 0.238 |
279
+ | **id** | 0.319 | 0.149 | 0.263 |
280
+ | **it** | 0.337 | 0.153 | 0.265 |
281
+ | **ja** | 0.276 | 0.141 | 0.241 |
282
+ | **pt** | 0.337 | 0.152 | 0.276 |
283
+ | **ru** | 0.298 | 0.124 | 0.251 |
284
+ | **vi** | 0.287 | 0.136 | 0.226 |
285
+ | **zh** | 0.302 | | 0.246 |
286
+
287
 
288
 
289
  ### Matryoshka Representation Benchmarks
290
 
291
  #### BEIR
292
 
293
+ | **NDCG@10** | **dim=128** | **dim=96** | **dim=64** |
294
+ |----------------|-------------|------------|------------|
295
+ | **avg** | 0.599 | 0.591 | 0.589 |
296
+ | **nfcorpus** | 0.346 | 0.340 | 0.347 |
297
+ | **fiqa** | 0.408 | 0.404 | 0.404 |
298
+ | **trec-covid** | 0.834 | 0.808 | 0.805 |
299
+ | **hotpotqa** | 0.766 | 0.764 | 0.756 |
300
+ | **nq** | 0.640 | 0.640 | 0.635 |
301
+
302
 
303
 
304
  #### MSMARCO
305
 
306
+ | **MRR@10** | **dim=128** | **dim=96** | **dim=64** |
307
+ |----------------|-------------|------------|------------|
308
+ | **msmarco** | 0.396 | 0.391 | 0.388 |
 
 
309
 
310
  ## Other Models
311