Improve Omni Nano embeddings with complete benchmark results

NOTICE

@@ -23,3 +23,5 @@ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
 SOFTWARE.
+
+Text component: avsolatorio/GIST-small-Embedding-v0, revision 75e62fd210b9fde790430e0b2f040b0b00a021b1. Its official model card declares MIT; the complete pinned card is retained in licenses/GIST-MODEL-CARD.md. The card identifies BAAI/bge-small-en-v1.5 as the upstream base and discloses its training data.

README.md

@@ -31,7 +31,7 @@ Text, images and speech in one embedding space. Vela Omni Nano supports multimod
 | Feature | Value |
 | --- | --- |
 | Modalities | Text, images, speech |
-| Total parameters | 134M (133,956,480) |
+| Total parameters | 144.6M (144,554,112) |
 | Embedding dimensions | 384 |
 | Text context | 512 tokens, including special tokens |
 | Audio input | Mono, 16 kHz, up to 30 seconds |
@@ -40,25 +40,20 @@ Text, images and speech in one embedding space. Vela Omni Nano supports multimod
 
 ## Evaluation
 
-Evaluated with MTEB 2.21.0. Scores use a 0–100 scale; higher is better. The full English and audio rows use mean task score; the three image rows are individual tasks. **Bold marks gains over the [previous Nano release](https://huggingface.co/llm-semantic-router/Vela-1.0-Omni-Nano/blob/70c3412bb0ddfb19386d673d4a102c753a574f1b/README.md).**
+MTEB 2.21.0; scores are 0–100. **Bold marks gains over the [previous Nano](https://huggingface.co/llm-semantic-router/Vela-1.0-Omni-Nano/blob/3a1efc48995e2b71622d65c6cf71cfc00bbf3aa3/README.md).** Both full panels retain every task and split.
 
 | Benchmark / metric | Previous Nano | Current Nano |
 | --- | ---: | ---: |
-| MTEB English v2 · all 41 tasks | 53.37 | 53.37 |
-| MAEB audio-only · all 19 tasks | 39.78 | 39.78 |
-| Oxford Pets · classification accuracy | 44.47 | **91.27** |
-| Tiny ImageNet · clustering NMI | 46.03 | **61.69** |
-| Oxford Pets · zero-shot accuracy | 15.26 | 11.64 |
+| English v2 · 41/41 · mean task | 53.37 | **64.88** |
+| English v2 · mean task type | 51.98 | **60.78** |
+| MAEB audio-only · 19/19 · mean task | 39.78 | 39.77 |
+| MAEB audio-only · 134 slots · mean task type | 46.97 | 46.97 |
 
-The update improves image classification, clustering and cross-modal retrieval at the same model size. Oxford Pets zero-shot accuracy decreases by 3.62 points; the text and audio components are unchanged.
-
-The [detailed evaluation](benchmarks/EVALUATION.md) contains task-type breakdowns, the selected 11-task text/image/audio panel, and routing and retrieval comparisons against the corresponding earlier model. **Gains over the named comparator are bold.** It also provides all regressions, sample counts and protocols.
-
-Routing evaluation uses labeled task adaptation; standard classification uses frozen-encoder probes. The English and audio panels do not constitute a complete multilingual MMTEB or image benchmark result. [Exact revisions and raw metrics](scores.json) accompany the task reports.
+Text and speech–text retrieval improve; image classification and clustering are preserved. Image-to-text retrieval and zero-shot Pets/SpeechCommands regress. See [complete scores, comparisons and methods](benchmarks/EVALUATION.md) for every task, sample count and tradeoff.
 
 ### Size and quality
 
-Nano reaches **65.25 V-measure** on ArXiv hierarchical clustering and **63.44 max AP** on NMSQA audio pair classification.
+Nano scores **64.89 ArXiv V-measure**, **63.28 NMSQA max AP** and **13.80 Vehicle V-measure**, at 144.6M total parameters.
 
 <table>
   <tr>
@@ -75,7 +70,7 @@ Select a figure to view it at full size.
 
 The dashed line marks the Pareto frontier among the displayed models: no displayed model is both smaller and at least as accurate, or at least as small and more accurate. The horizontal axis uses **total model parameters on a logarithmic scale**, including all three modalities for Vela.
 
-The figures retain reported leaderboard points from **September 17, 2026** and add a separately marked [F2LLM v2 80M rerun](benchmarks/peer-evaluations/f2llm-v2-80m-arxiv.json) at **69.74 ArXiv V-measure**. It exceeds both Nano’s 65.25 and Mini’s 65.52 on this task; neither Vela model is on the updated ArXiv frontier. The rerun uses the same official task and data, with F2’s task instruction, BF16 and 8192-token limit; other peer protocols also vary. These task-level comparisons do not establish overall benchmark leadership. [All four size–quality figures](benchmarks/pareto-gallery.md) · [Methodology and every plotted point](benchmarks/pareto-methodology.md).
+The figures retain the September 17, 2026 registry snapshot and separate measured peer observations. [F2 80M](benchmarks/peer-evaluations/f2llm-v2-80m-arxiv.json) reaches **69.74** ArXiv V-measure; [AST 86.59M](benchmarks/peer-evaluations/ast-vehicle.json) reaches **14.34** Vehicle V-measure. Nano is on neither updated frontier. Reported and measured protocols differ, and these tasks do not establish overall benchmark leadership. [Every point and protocol](benchmarks/pareto-methodology.md).
 
 ## Usage
 
@@ -101,7 +96,7 @@ Pass a list of Pillow images to `model.encode_image(images)`. Pass a list of mon
 
 ## Training and license
 
-Cross-modal alignment uses COCO image–caption pairs from the CC-BY 2.0 image subset and [LibriSpeech](https://www.openslr.org/12/) speech–transcript pairs (CC-BY 4.0). COCO annotations, Banking77, and MASSIVE are provided under CC-BY 4.0.
+Cross-modal alignment uses COCO image–caption pairs from the CC-BY 2.0 image subset and [LibriSpeech](https://www.openslr.org/12/) speech–transcript pairs (CC-BY 4.0). COCO annotations are CC-BY 4.0. The frozen [GIST text backbone](https://huggingface.co/avsolatorio/GIST-small-Embedding-v0) is MIT-licensed; its pinned model card and component notices are included.
 
 The repository includes the native model code, component configurations, and tokenizer and processor files. See [NOTICE](./NOTICE) and [LICENSE](./LICENSE) for component attribution and license terms.
 

benchmarks/EVALUATION.md

@@ -1,115 +1,155 @@
-# Vela Omni Nano — Evaluation
-
-[Back to model card](../README.md) · [Raw scores](../scores.json)
-
-## Task-adapted routing and retrieval
-
-Scores use a 0–100 scale; higher is better. Both models use the same examples and retrieval pools. Bold Vela scores improve on multi-modal-embed-small.
-
-| Metric | [multi-modal-embed-small](https://huggingface.co/llm-semantic-router/multi-modal-embed-small) | Vela Omni Nano |
-| --- | ---: | ---: |
-| Banking77 · Accuracy | 70.42 | **76.82** |
-| Banking77 · Macro-F1 | 70.21 | **76.58** |
-| MASSIVE English · Accuracy | 65.95 | **69.35** |
-| MASSIVE English · Macro-F1 | 61.72 | **65.24** |
-| COCO · Image → text · R@1 | 40.83 | **65.61** |
-| COCO · Image → text · R@5 | 67.19 | **88.82** |
-| COCO · Image → text · R@10 | 78.49 | **96.11** |
-| COCO · Text → image · R@1 | 30.18 | **53.07** |
-| COCO · Text → image · R@5 | 59.42 | **84.20** |
-| COCO · Text → image · R@10 | 74.29 | **93.49** |
-| LibriSpeech · Audio → text · R@1 | 4.21 | **9.00** |
-| LibriSpeech · Audio → text · R@5 | 11.99 | **22.29** |
-| LibriSpeech · Audio → text · R@10 | 19.03 | **31.48** |
-| LibriSpeech · Text → audio · R@1 | 9.58 | **14.02** |
-| LibriSpeech · Text → audio · R@5 | 22.53 | **31.69** |
-| LibriSpeech · Text → audio · R@10 | 30.69 | **40.42** |
-
-Macro-F1 gives equal weight to every intent class (77 for Banking77 and 60 for MASSIVE), complementing the query-weighted accuracy; undefined class F1 is zero. Text evaluation uses fixed class prototypes: 3,080 Banking77 and 2,972 MASSIVE English queries. Vela Omni is adapted using training examples and intent labels from these two datasets; the baseline is evaluated as published without additional adaptation here. These scores measure supervised task adaptation with nearest-prototype classification, not zero-shot transfer or the MTEB classification protocol.
-
-Image retrieval uses the COCO Karpathy CC-BY2 image subset: 823 images and 4,115 captions. Speech retrieval uses 2,611 LibriSpeech test-clean clips and 2,610 unique transcripts, with clips limited to 30 seconds. Recall is measured against each complete candidate pool, accepting all matching positives.
-
-Evaluation uses FP32 embeddings, cosine similarity, and a common 128-token text limit. [Full scores and protocol](../scores.json) include model revisions, sample counts, and paired uncertainty against the original and previous models. These fixed evaluation pools are reused across releases.
-
-## Release comparison
-
-Compared with the [previous Nano release](https://huggingface.co/llm-semantic-router/Vela-1.0-Omni-Nano/blob/70c3412bb0ddfb19386d673d4a102c753a574f1b/README.md), the parameter count remains 133,956,480 and the public input limits are unchanged. **Bold values improve on that release.**
-
-| Measured component | Previous Nano | Current Nano | Change |
-| --- | ---: | ---: | ---: |
-| Routing / cross-modal retrieval | 54.82 | **59.37** | +4.55 |
-| English v2 · mean task type | 51.98 | 51.98 | +0.00 |
-| Three image tasks · mean | 35.26 | **54.87** | +19.61 |
-| Audio-only · mean task type | 46.97 | 46.97 | +0.00 |
-
-The routing/retrieval component averages the two text accuracies, all six image recalls and all six audio recalls within their families, then weights the three families equally. The image component averages Oxford Pets classification accuracy, Tiny ImageNet NMI and Oxford Pets zero-shot accuracy. The English and audio rows use their complete official task-type means.
-
-A product release comparison weights these components 30%, 30%, 20% and 20%, respectively, yielding **+5.29 points**. This is a product-specific comparison, not an official benchmark aggregate or SOTA rank. Every underlying score and the paired retrieval uncertainty are in [scores.json](../scores.json).
-
-Oxford Pets classification improves by **46.80 points**, Tiny ImageNet NMI by **15.66 points**, and zero-shot Oxford Pets accuracy decreases by **3.62 points**. All six COCO recalls improve by 10.21–18.35 points. The text and audio computations are byte-identical to the previous release; their complete 41-task and 19-task reports retain the original evaluated identity and document [applicability to the current model](component-equivalence.json).
-
-## Standard benchmarks
-
-Selected text, image and audio tasks evaluated with [MTEB 2.21.0](https://github.com/embeddings-benchmark/mteb), frozen encoders and official task protocols. Scores use a 0–100 scale; higher is better. **Bold marks the higher score between the two current Vela models.** This selected panel is not a complete MTEB, MMTEB, MIEB or MAEB result.
-
-| Task | Metric | Omni Nano | Omni Mini |
-| --- | --- | ---: | ---: |
-| SICK-R | Spearman ρ | 71.70 | **76.02** |
-| STSBenchmark | Spearman ρ | **77.92** | 75.25 |
-| ArguAna | nDCG@10 | **37.51** | 36.41 |
-| 20 Newsgroups · Clustering v2 | V-measure | **44.88** | 39.56 |
-| MTOP domain · English | Accuracy | **86.72** | 86.00 |
-| Oxford Pets · Classification | Accuracy | 91.27 | **92.26** |
-| Tiny ImageNet · Clustering | NMI | 61.69 | **75.55** |
-| Oxford Pets · Zero-shot | Accuracy | **11.64** | 9.57 |
-| CREMA-D · Classification | Accuracy | **29.76** | 21.70 |
-| CREMA-D · Clustering | V-measure | **2.36** | 0.56 |
-| Speech Commands v0.02 · Zero-shot | Accuracy | 14.97 | **42.00** |
-
-Evaluation uses FP32, batch size 8 and seed 20260918. Text limits are 512 tokens for Nano and 32,768 for Mini. The benchmark adapter truncates longer inputs: on ArguAna this affects 222 of 10,080 Nano inputs and none for Mini. Text, image and audio classification use task-specific labeled probes; zero-shot tasks compare media with fixed text labels. CREMA-D classification uses the official five-fold protocol on its named `train` split, with the encoder frozen. Official clustering subsampling is preserved.
-
-[Full metrics and protocol](../scores.json) include dataset and model revisions, evaluation counts, repeated-probe results, classification F1/precision/recall, retrieval metrics and clustering statistics. MTOP validation results are included there separately. These measurements show that fine-grained image label matching and audio emotion recognition need stronger alignment; speech retrieval scores should not be read as general audio understanding.
-
-For the previous Nano release, on one AMD gfx942 GPU, synthetic batches of eight texts took 4.28 ms median at 128 tokens and 12.45 ms at 512 tokens (FP32, SDPA; 20 measured batches). Longer inputs increase latency and memory; [measurement details](../scores.json) include p95 latency and peak memory. These historical timings exclude model loading and are not service latency; this release does not add a new timing measurement.
-
-## MTEB English v2
-
-Complete **41-task English v2** panel across seven task types.
-
-| Task type | Tasks | Vela Omni Nano | Vela Omni Mini |
-|---|---:|---:|---:|
-| Classification | 8 | 65.53 | **70.38** |
-| Clustering | 8 | **47.07** | 45.10 |
-| Pair classification | 3 | **74.95** | 73.54 |
-| Reranking | 2 | **45.57** | 43.48 |
-| Retrieval | 10 | **29.02** | 28.42 |
-| Semantic similarity | 9 | **72.45** | 72.38 |
-| Summarization | 1 | **29.25** | 28.00 |
-| Mean task | 41 | 53.37 | **53.54** |
-| Mean task type | 7 | **51.98** | 51.61 |
-
-Scores ×100; **bold marks the higher current Vela score**. MTEB 2.21.0; FP32, batch size 8, seed 20260918, no query instructions. Native token limits: Nano 512; Mini 32,768. Training includes Banking77 and MASSIVE, so results include task-adapted behavior.
-
-[All 41 Nano task results and detailed metrics](mteb-eng-v2.json) and [Mini results](https://huggingface.co/llm-semantic-router/Vela-1.0-Omni-Mini/blob/main/benchmarks/mteb-eng-v2.json) include exact evaluated revisions. The current models retain the evaluated text weights and processing. The English panel does not measure multilingual, image or audio quality.
-
-## Audio benchmarks
-
-Complete MAEB(beta) audio-only panel: **19 tasks and 134 subset/split combinations**, evaluated with MTEB 2.21.0.
-
-| Official task group | Vela Omni Nano | Vela Omni Mini |
-|---|---:|---:|
-| Classification | 39.57 | **42.95** |
-| Clustering | **5.42** | 3.12 |
-| Pair classification | 57.19 | **62.90** |
-| Retrieval | **68.90** | 53.38 |
-| Reranking | **63.79** | 63.26 |
-| Mean task | 39.78 | **41.43** |
-| Mean task type | **46.97** | 45.12 |
-
-Scores ×100; **bold marks the higher current Vela score**. Each group follows the official mean of task scores. Classification includes official supervised probes; joint text/audio tasks are excluded.
-
-[All 19 task metrics and exact evaluated revisions](maeb-audio-only.json). The current models retain the evaluated audio weights and processing.
-
-## Size–quality comparisons
-
-See the [four-figure gallery](pareto-gallery.md) and [Pareto methodology and data](pareto-methodology.md) for the task selection, complete plotted peer set, parameter definitions and source snapshot.
+# Evaluation details
+
+This report identifies Nano artifact `96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4` (144,554,112 parameters). The previous Nano is [3a1efc48](https://huggingface.co/llm-semantic-router/Vela-1.0-Omni-Nano/tree/3a1efc48995e2b71622d65c6cf71cfc00bbf3aa3) (`fb9cd39c9980a085b75456c715c2295b997784fe733314be68c13635f6384edf`). Scores are shown on a 0–100 scale. Bold means an unrounded improvement over the explicitly named comparator.
+
+## Complete panels
+
+English contains all 41 tasks and seven task types. Audio contains all 19 audio-only MAEB tasks, five task types and 134 subset/split slots; this is not the complete joint-modality MAEB panel. Official aggregation averages tasks or task types, not raw subset counts.
+
+| Panel | Previous mean task | Current mean task | Previous mean type | Current mean type |
+| --- | ---: | ---: | ---: | ---: |
+| English v2 · 41/41 | 53.37 | **64.88** | 51.98 | **60.78** |
+| MAEB audio-only · 19/19 | 39.78 | 39.77 | 46.97 | 46.97 |
+
+English was evaluated on an earlier native artifact with exactly the same 199 GIST text tensors and identical source/configuration/tokenizer bytes. Its actual evaluated identity is retained in [the full report](mteb-eng-v2.json) and [text equivalence evidence](component-equivalence.json). The current image, audio and cross-modal scores use fresh current-artifact inference.
+
+### All English tasks
+
+| Task | Metric | Previous | Current | Change (pp) |
+| --- | --- | ---: | ---: | ---: |
+| ArguAna | ndcg_at_10 | 37.5140 | **59.2460** | +21.7320 |
+| ArXivHierarchicalClusteringP2P | v_measure | 65.2540 | 64.8885 | -0.3655 |
+| ArXivHierarchicalClusteringS2S | v_measure | 55.9895 | **57.4432** | +1.4536 |
+| AskUbuntuDupQuestions | map_at_1000 | 60.5140 | **62.3290** | +1.8150 |
+| BIOSSES | cosine_spearman | 62.2454 | **86.9875** | +24.7421 |
+| Banking77Classification | accuracy | 69.5747 | **82.1429** | +12.5682 |
+| BiorxivClusteringP2P.v2 | v_measure | 39.8292 | **41.3138** | +1.4846 |
+| CQADupstackGamingRetrieval | ndcg_at_10 | 42.9760 | **56.9790** | +14.0030 |
+| CQADupstackUnixRetrieval | ndcg_at_10 | 30.3620 | **39.5860** | +9.2240 |
+| ClimateFEVERHardNegatives | ndcg_at_10 | 16.0740 | **31.8460** | +15.7720 |
+| FEVERHardNegatives | ndcg_at_10 | 24.0550 | **87.5760** | +63.5210 |
+| FiQA2018 | ndcg_at_10 | 22.5880 | **39.1430** | +16.5550 |
+| HotpotQAHardNegatives | ndcg_at_10 | 26.9870 | **66.3490** | +39.3620 |
+| ImdbClassification | accuracy | 62.1364 | **91.9476** | +29.8112 |
+| MTOPDomainClassification | accuracy | 86.7191 | **94.9179** | +8.1988 |
+| MassiveIntentClassification | accuracy | 60.3934 | **70.9684** | +10.5750 |
+| MassiveScenarioClassification | accuracy | 73.3726 | **76.1130** | +2.7404 |
+| MedrxivClusteringP2P.v2 | v_measure | 37.7681 | **39.8302** | +2.0621 |
+| MedrxivClusteringS2S.v2 | v_measure | 36.1335 | **37.8751** | +1.7416 |
+| MindSmallReranking | max_over_subqueries_map_at_1000 | 30.6230 | **32.3660** | +1.7430 |
+| SCIDOCS | ndcg_at_10 | 16.8270 | **21.8890** | +5.0620 |
+| SICK-R | cosine_spearman | 71.6962 | **80.5317** | +8.8356 |
+| STS12 | cosine_spearman | 68.1692 | **75.5659** | +7.3967 |
+| STS13 | cosine_spearman | 74.4034 | **86.2635** | +11.8600 |
+| STS14 | cosine_spearman | 68.0265 | **82.2988** | +14.2723 |
+| STS15 | cosine_spearman | 77.8614 | **88.7365** | +10.8750 |
+| STSBenchmark | cosine_spearman | 77.9240 | **87.0782** | +9.1542 |
+| SprintDuplicateQuestions | max_ap | 89.7910 | **95.7996** | +6.0087 |
+| StackExchangeClustering.v2 | v_measure | 57.3446 | **58.4244** | +1.0798 |
+| StackExchangeClusteringP2P.v2 | v_measure | 39.3767 | **41.0530** | +1.6763 |
+| TRECCOVID | ndcg_at_10 | 41.2920 | **69.1210** | +27.8290 |
+| Touche2020Retrieval.v3 | ndcg_at_10 | 31.4770 | **48.3630** | +16.8860 |
+| ToxicConversationsClassification | accuracy | 64.4092 | **71.9043** | +7.4951 |
+| TweetSentimentExtractionClassification | accuracy | 50.7782 | **62.7929** | +12.0147 |
+| TwentyNewsgroupsClustering.v2 | v_measure | 44.8847 | **50.8726** | +5.9879 |
+| TwitterSemEval2015 | max_ap | 56.3448 | **72.9515** | +16.6068 |
+| TwitterURLCorpus | max_ap | 78.7021 | **85.3034** | +6.6013 |
+| SummEvalSummarization.v2 | cosine_spearman | 29.2496 | **31.8344** | +2.5848 |
+| AmazonCounterfactualClassification | accuracy | 56.8209 | **71.8507** | +15.0299 |
+| STS17 | cosine_spearman | 84.1659 | **89.0210** | +4.8551 |
+| STS22.v2 | cosine_spearman | 67.5701 | **68.7533** | +1.1833 |
+
+### All audio tasks
+
+| Task | Metric | Previous | Current | Change (pp) |
+| --- | --- | ---: | ---: | ---: |
+| JamAltArtistA2ARetrieval | ndcg_at_10 | 68.9005 | 68.9005 | +0.0000 |
+| BeijingOpera | accuracy | 68.6791 | 68.6791 | +0.0000 |
+| BirdCLEF | accuracy | 12.9000 | 12.9000 | +0.0000 |
+| CREMA_D | accuracy | 29.7637 | 29.7637 | +0.0000 |
+| CommonLanguageAgeDetection | accuracy | 16.0400 | 16.0400 | +0.0000 |
+| GTZANGenre | accuracy | 50.3000 | 50.3000 | +0.0000 |
+| IEMOCAPGender | accuracy | 58.4519 | 58.4519 | +0.0000 |
+| MInDS14 | accuracy | 23.3212 | 23.3212 | +0.0000 |
+| MridinghamTonic | accuracy | 32.5066 | 32.3632 | -0.1434 |
+| SIBFLEURS | accuracy | 18.3047 | 18.3047 | +0.0000 |
+| VoxCelebSA | accuracy | 33.1691 | 33.1691 | +0.0000 |
+| VoxPopuliLanguageID | accuracy | 91.8000 | 91.8000 | +0.0000 |
+| CREMA_DClustering | v_measure | 2.3602 | 2.3602 | +0.0000 |
+| VehicleSoundClustering | v_measure | 13.8007 | 13.8007 | +0.0000 |
+| VoxPopuliGenderClustering | v_measure | 0.1045 | 0.1045 | +0.0000 |
+| CREMADPairClassification | max_ap | 54.0802 | **54.2012** | +0.1210 |
+| NMSQAPairClassification | max_ap | 63.4355 | 63.2834 | -0.1521 |
+| VoxPopuliAccentPairClassification | max_ap | 54.0504 | 54.0220 | -0.0284 |
+| GTZANAudioReranking | map_at_1000 | 63.7940 | 63.7940 | +0.0000 |
+
+Rich metrics, all repetitions and every subset/split are in [English JSON](mteb-eng-v2.json) and [audio JSON](maeb-audio-only.json). Auxiliary statistics that are undefined are null; primary scores are finite.
+
+## Routing and cross-modal retrieval
+
+This fixed common protocol uses 128-token text inputs, fresh labeled training prototypes for classification, and cosine ranking with complete positive associations for retrieval. It is distinct from official MTEB classification. The original small comparator is task-adapted; the current text backbone is frozen GIST. Evaluation rows are reused across releases, so these are known-set progress measurements, not a new blind test.
+
+| Metric | Original small | Current Nano | Previous Nano | Change vs previous (pp) |
+| --- | ---: | ---: | ---: | ---: |
+| audio.audio_to_text.recall@1 | 4.21 | **17.16** | 9.00 | +8.1578 |
+| audio.audio_to_text.recall@10 | 19.03 | **46.99** | 31.48 | +15.5113 |
+| audio.audio_to_text.recall@5 | 11.99 | **36.65** | 22.29 | +14.3623 |
+| audio.text_to_audio.recall@1 | 9.58 | **25.44** | 14.02 | +11.4176 |
+| audio.text_to_audio.recall@10 | 30.69 | **55.90** | 40.42 | +15.4789 |
+| audio.text_to_audio.recall@5 | 22.53 | **46.13** | 31.69 | +14.4444 |
+| image.image_to_text.recall@1 | 40.83 | **60.87** | 65.61 | -4.7388 |
+| image.image_to_text.recall@10 | 78.49 | **94.90** | 96.11 | -1.2151 |
+| image.image_to_text.recall@5 | 67.19 | **87.12** | 88.82 | -1.7011 |
+| image.text_to_image.recall@1 | 30.18 | **55.82** | 53.07 | +2.7461 |
+| image.text_to_image.recall@10 | 74.29 | **94.26** | 93.49 | +0.7776 |
+| image.text_to_image.recall@5 | 59.42 | **85.22** | 84.20 | +1.0207 |
+| text.banking77.accuracy | 70.42 | **87.99** | 76.82 | +11.1688 |
+| text.massive-en.accuracy | 65.95 | **81.97** | 69.35 | +12.6178 |
+| text.banking77.macro_f1 | 70.21 | **87.92** | 76.58 | +11.3326 |
+| text.massive-en.macro_f1 | 61.72 | **79.23** | 65.24 | +13.9885 |
+
+Bold in this common table compares Nano with its original small counterpart. Exact numerators/denominators, 77 Banking classes and 60 MASSIVE classes, Macro-F1 definitions and all paired intervals are in [scores.json](../scores.json). The original paired utility uses two primary metrics per family; its interval is not an interval for the all-14 common mean or the product aggregate below.
+
+## Fixed text/image/audio panel
+
+Eleven official tasks, twelve reported split rows. MTOP validation is descriptive and is not an extra task in the aggregate. No prompts or checkpoints were selected using these scores.
+
+| Task / split | Main metric | Previous Nano | Current Nano | Companion Mini |
+| --- | --- | ---: | ---: | ---: |
+| SICK-R / test | cosine_spearman | 71.70 | **80.53** | 81.46 |
+| STSBenchmark / test | cosine_spearman | 77.92 | **87.08** | 84.78 |
+| ArguAna / test | ndcg_at_10 | 37.51 | **59.25** | 68.21 |
+| TwentyNewsgroupsClustering.v2 / test | v_measure | 44.88 | **50.87** | 48.54 |
+| MTOPDomainClassification / validation | accuracy | 87.59 | **94.77** | 91.25 |
+| MTOPDomainClassification / test | accuracy | 86.72 | **94.92** | 91.40 |
+| OxfordPets / test | accuracy | 91.27 | 91.27 | 92.26 |
+| TinyImageNetClustering / valid | nmi | 61.69 | 61.69 | 75.55 |
+| OxfordPetsZeroShot / test | accuracy | 11.64 | 8.01 | 12.65 |
+| CREMA_D / train | accuracy | 29.76 | 29.76 | 21.80 |
+| CREMA_DClustering / train | v_measure | 2.36 | 2.36 | 0.54 |
+| SpeechCommandsZeroshotv0.02 / test | accuracy | 14.97 | 14.11 | 42.00 |
+
+The model identities and actual encoding observations are included in [scores.json](../scores.json). Bold in this panel compares Nano with previous Nano, not with Mini.
+
+## Progress and tradeoffs
+
+| Component | Weight | Previous | Current | Change (pp) |
+| --- | ---: | ---: | ---: | ---: |
+| common14 | 30% | 59.3729 | 67.5741 | +8.2012 |
+| English41_MeanTaskType | 30% | 51.9755 | 60.7818 | +8.8063 |
+| image3 | 20% | 54.8658 | 53.6575 | -1.2083 |
+| audio19_MeanTaskType | 20% | 46.9744 | 46.9678 | -0.0066 |
+
+Weighted product progress is **+4.8593 points**. This is a product comparison, not an official leaderboard aggregate or overall SOTA claim. The common component equally weights text, image and audio families and uses all 14 metrics. The image component equally averages Pets classification, TinyImageNet NMI and Pets zero-shot accuracy.
+
+Regressions remain visible: all three image-to-text recall ranks decline; Pets zero-shot decreases 3.625 points; SpeechCommands zero-shot decreases 0.859; ArXiv P2P decreases 0.3655. Three audio-panel tasks decline slightly (MridinghamTonic, NMSQA and VoxPopuliAccent); all other changes are listed above. Classification/clustering preservation does not imply preservation of cross-modal or short-label zero-shot behavior.
+
+## Training, evaluation and cost limits
+
+The GIST foundation card discloses MTEB classification training data. We do not claim absence of pretraining overlap. Cross-modal optimization uses COCO and LibriSpeech TRAIN only with complete positives and no benchmark/Studio-example training. One final 300-step pair of orthogonal media rotations was exported; text and media encoders are frozen.
+
+The current model has 144,554,112 parameters and 384-dimensional FP32 vectors (1,536 raw bytes per vector, excluding index overhead). No current-artifact latency or peak-memory claim is made. Historical measurements in scores.json retain their measured artifact identity.
+
+## Task-level Pareto comparisons
+
+[Every plotted observation](pareto-data.json) and [methodology](pareto-methodology.md) retain the reported API snapshot and separate independent peer reruns. F2 reaches 69.7422 ArXiv V-measure and AST reaches 14.3429 Vehicle V-measure. Nano is not on either updated frontier. Parameter counts refer to whole stored models, including all Vela modalities. These four tasks do not establish overall size SOTA.

benchmarks/component-equivalence.json

@@ -1,30 +1,30 @@
 {
   "schema_version": 1,
   "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
-  "parameters": 133956480,
+  "parameters": 144554112,
   "evaluated_model": {
-    "revision": "70c3412bb0ddfb19386d673d4a102c753a574f1b",
-    "native_artifact_sha256": "b596f1dbab89369555e0a72a0d6854d780e305686750949b308c8f167993bdeb"
+    "model": "local/Vela-Omni-Nano-GIST-Rotation",
+    "revision": "29994142677559ca657c27b657eda0dddbf583bd4cf42a3a31095ab3f99d1646",
+    "revision_kind": "local native artifact fingerprint, not a Hugging Face commit",
+    "native_artifact_sha256": "29994142677559ca657c27b657eda0dddbf583bd4cf42a3a31095ab3f99d1646"
   },
   "applies_to": {
     "revision": "self",
-    "native_artifact_sha256": "fb9cd39c9980a085b75456c715c2295b997784fe733314be68c13635f6384edf"
+    "native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4"
   },
   "unchanged_components": [
-    "text",
-    "audio"
+    "text"
   ],
-  "method": "Exact stored tensor values, dtypes and shapes, with byte-identical inference source, configuration, tokenizer and media processors. The official evaluator, runtime, seed and batching remain the same.",
   "tensor_identity": {
-    "text_tensors": 105,
-    "audio_tensors": 85,
-    "all_nonimage_tensors": 243,
-    "exact": true
+    "text_tensors": 199,
+    "text_parameters": 33360000,
+    "stored_values_dtypes_shapes_exact": true,
+    "effective_float32_tensors_exact": true
   },
   "inference_files_exact": true,
+  "method": "Both complete native packages were rehashed; all text tensors and inference source, configuration and tokenizer bytes match. Official evaluator, runtime, seed, batching, CLS pooling and 512-token limit are identical.",
   "reports": [
-    "benchmarks/mteb-eng-v2.json",
-    "benchmarks/maeb-audio-only.json"
+    "benchmarks/mteb-eng-v2.json"
   ],
-  "scope": "The original reports retain their actual evaluated revision and whole-model fingerprint. Applicability follows unchanged text/audio computation; these full panels were not rerun on the updated image artifact. Image and cross-modal scores use fresh evaluations."
+  "scope": "The full English report keeps the artifact that was actually evaluated. Only text-only scores apply through this bridge. Image, audio and cross-modal results are freshly evaluated on the current artifact."
 }

benchmarks/maeb-audio-only.json

@@ -1,8 +1,8 @@
 {
   "schema_version": 1,
   "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
-  "evaluated_revision": "70c3412bb0ddfb19386d673d4a102c753a574f1b",
-  "native_artifact_sha256": "b596f1dbab89369555e0a72a0d6854d780e305686750949b308c8f167993bdeb",
+  "evaluated_revision": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
+  "native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
   "benchmark": "MAEB(beta, audio-only)",
   "framework": "mteb",
   "framework_version": "2.21.0",
@@ -18,12 +18,12 @@
   "aggregation": {
     "official": {
       "Any2AnyRetrieval": 0.6890050000000001,
-      "AudioClassification": 0.39566923071716426,
+      "AudioClassification": 0.39553889510295764,
       "AudioClustering": 0.054218008138417706,
-      "AudioPairClassification": 0.571886979184418,
+      "AudioPairClassification": 0.5716888010864635,
       "AudioReranking": 0.63794,
-      "Mean(Task)": 0.39776955262406916,
-      "Mean(TaskType)": 0.46974384360800003
+      "Mean(Task)": 0.3976628038845884,
+      "Mean(TaskType)": 0.4696781408655678
     },
     "definition": "Official TaskResult aggregation within each task, followed by official mean_task, mean_task_type and task_types aggregation. The 134 subset/split slots are not averaged as independent tasks.",
     "score_scale": "0-1; multiply by 100 for display"
@@ -93,26 +93,26 @@
             "mrr_at_3": 1.0,
             "mrr_at_5": 1.0,
             "mteb_version": "2.21.0",
-            "nauc_map_at_1000_diff1": -0.26823787167124846,
-            "nauc_map_at_1000_max": 0.03606710761894326,
-            "nauc_map_at_1000_std": 0.08403852972987866,
-            "nauc_map_at_100_diff1": -0.23432744991537238,
-            "nauc_map_at_100_max": 0.035668155835282045,
-            "nauc_map_at_100_std": -0.0006452261766109746,
+            "nauc_map_at_1000_diff1": -0.26823840584449077,
+            "nauc_map_at_1000_max": -0.06625526205504577,
+            "nauc_map_at_1000_std": 0.0840383580140474,
+            "nauc_map_at_100_diff1": -0.23432737789937702,
+            "nauc_map_at_100_max": -0.07948367353583083,
+            "nauc_map_at_100_std": -0.0006448956579781191,
             "nauc_map_at_10_diff1": -0.2869264081067235,
-            "nauc_map_at_10_max": 0.029328117059972198,
+            "nauc_map_at_10_max": -0.0836560830061231,
             "nauc_map_at_10_std": -0.04345768313582453,
             "nauc_map_at_1_diff1": 0.20980634933775819,
-            "nauc_map_at_1_max": -0.04304592277045689,
+            "nauc_map_at_1_max": -0.09263910114107297,
             "nauc_map_at_1_std": -0.22340209123623483,
             "nauc_map_at_20_diff1": -0.280837964753022,
-            "nauc_map_at_20_max": 0.02741026318933178,
+            "nauc_map_at_20_max": -0.08055764307462374,
             "nauc_map_at_20_std": -0.059044453361231215,
             "nauc_map_at_3_diff1": -0.13917780935018328,
-            "nauc_map_at_3_max": 0.008404697556102434,
+            "nauc_map_at_3_max": -0.06877420959425827,
             "nauc_map_at_3_std": -0.11008707075398086,
             "nauc_map_at_5_diff1": -0.22234226882082003,
-            "nauc_map_at_5_max": 0.011297272220205117,
+            "nauc_map_at_5_max": -0.06592771530072024,
             "nauc_map_at_5_std": -0.049123829077376946,
             "nauc_mrr_at_1000_diff1": null,
             "nauc_mrr_at_1000_max": null,
@@ -135,68 +135,68 @@
             "nauc_mrr_at_5_diff1": null,
             "nauc_mrr_at_5_max": null,
             "nauc_mrr_at_5_std": null,
-            "nauc_ndcg_at_1000_diff1": -0.3337214782017371,
-            "nauc_ndcg_at_1000_max": 0.04759085334619801,
-            "nauc_ndcg_at_1000_std": 0.122873772807387,
-            "nauc_ndcg_at_100_diff1": -0.17380061097013183,
-            "nauc_ndcg_at_100_max": 0.041142794769809574,
-            "nauc_ndcg_at_100_std": -0.03434859536569615,
-            "nauc_ndcg_at_10_diff1": -0.46497757081401486,
-            "nauc_ndcg_at_10_max": 0.04815445390119436,
+            "nauc_ndcg_at_1000_diff1": -0.33372165083689326,
+            "nauc_ndcg_at_1000_max": -0.04645831342162314,
+            "nauc_ndcg_at_1000_std": 0.12287360101720309,
+            "nauc_ndcg_at_100_diff1": -0.17380035615696104,
+            "nauc_ndcg_at_100_max": -0.09027876790780465,
+            "nauc_ndcg_at_100_std": -0.03434839843928912,
+            "nauc_ndcg_at_10_diff1": -0.4649775708140152,
+            "nauc_ndcg_at_10_max": -0.035494014369584236,
             "nauc_ndcg_at_10_std": 0.13129800035275796,
             "nauc_ndcg_at_1_diff1": null,
             "nauc_ndcg_at_1_max": null,
             "nauc_ndcg_at_1_std": null,
             "nauc_ndcg_at_20_diff1": -0.401398780178619,
-            "nauc_ndcg_at_20_max": 0.04280734030262537,
+            "nauc_ndcg_at_20_max": -0.03786746450816836,
             "nauc_ndcg_at_20_std": 0.09962158190188782,
             "nauc_ndcg_at_3_diff1": -0.6253490259277262,
-            "nauc_ndcg_at_3_max": 0.048105273750879904,
+            "nauc_ndcg_at_3_max": -0.0032353632113252513,
             "nauc_ndcg_at_3_std": 0.1359742585981673,
-            "nauc_ndcg_at_5_diff1": -0.5371449887722016,
-            "nauc_ndcg_at_5_max": 0.03889141668980628,
+            "nauc_ndcg_at_5_diff1": -0.5371449887722024,
+            "nauc_ndcg_at_5_max": -0.010623457026757448,
             "nauc_ndcg_at_5_std": 0.1556458040170057,
             "nauc_precision_at_1000_diff1": -0.24112362913109234,
-            "nauc_precision_at_1000_max": 0.0092717373635064,
+            "nauc_precision_at_1000_max": 0.06022247402355004,
             "nauc_precision_at_1000_std": 0.336005021890332,
             "nauc_precision_at_100_diff1": -0.20150434054621924,
-            "nauc_precision_at_100_max": 0.04296562186990787,
+            "nauc_precision_at_100_max": -0.04088858982721537,
             "nauc_precision_at_100_std": 0.16859627895436816,
             "nauc_precision_at_10_diff1": -0.43129017145659837,
-            "nauc_precision_at_10_max": 0.04979040717358857,
+            "nauc_precision_at_10_max": -0.04029572520596633,
             "nauc_precision_at_10_std": 0.12605961887140518,
             "nauc_precision_at_1_diff1": null,
             "nauc_precision_at_1_max": null,
             "nauc_precision_at_1_std": null,
             "nauc_precision_at_20_diff1": -0.3492059840917737,
-            "nauc_precision_at_20_max": 0.042085627797694765,
+            "nauc_precision_at_20_max": -0.034896129699382744,
             "nauc_precision_at_20_std": 0.07817873379205106,
             "nauc_precision_at_3_diff1": -0.613214253424573,
-            "nauc_precision_at_3_max": 0.050657650672696494,
+            "nauc_precision_at_3_max": 0.003070756563273359,
             "nauc_precision_at_3_std": 0.13951460409864888,
             "nauc_precision_at_5_diff1": -0.5135399238274392,
-            "nauc_precision_at_5_max": 0.038303431348652515,
+            "nauc_precision_at_5_max": -0.00871318651044832,
             "nauc_precision_at_5_std": 0.1593691155832114,
             "nauc_recall_at_1000_diff1": null,
             "nauc_recall_at_1000_max": null,
             "nauc_recall_at_1000_std": null,
             "nauc_recall_at_100_diff1": -0.043642561781925035,
-            "nauc_recall_at_100_max": 0.0348804671475377,
+            "nauc_recall_at_100_max": -0.10221943795434527,
             "nauc_recall_at_100_std": -0.06999989904295278,
             "nauc_recall_at_10_diff1": -0.2223971636720221,
-            "nauc_recall_at_10_max": 0.028946155281078068,
+            "nauc_recall_at_10_max": -0.09106876991880855,
             "nauc_recall_at_10_std": -0.06680015976295517,
             "nauc_recall_at_1_diff1": 0.20980634933775819,
-            "nauc_recall_at_1_max": -0.04304592277045689,
+            "nauc_recall_at_1_max": -0.09263910114107297,
             "nauc_recall_at_1_std": -0.22340209123623483,
             "nauc_recall_at_20_diff1": -0.18021340281671802,
-            "nauc_recall_at_20_max": 0.015823083765733933,
+            "nauc_recall_at_20_max": -0.08735815621877004,
             "nauc_recall_at_20_std": -0.10053789294076933,
             "nauc_recall_at_3_diff1": -0.12162904087705163,
-            "nauc_recall_at_3_max": 0.006917175953984682,
+            "nauc_recall_at_3_max": -0.06568270251238352,
             "nauc_recall_at_3_std": -0.11636712576289739,
             "nauc_recall_at_5_diff1": -0.1872174363939106,
-            "nauc_recall_at_5_max": 0.006370384685177946,
+            "nauc_recall_at_5_max": -0.07173991565116111,
             "nauc_recall_at_5_std": -0.06164080737125537,
             "ndcg_at_1": 1.0,
             "ndcg_at_10": 0.61555,
@@ -249,26 +249,26 @@
             "mrr_at_3": 1.0,
             "mrr_at_5": 1.0,
             "mteb_version": "2.21.0",
-            "nauc_map_at_1000_diff1": -0.30630348524392814,
-            "nauc_map_at_1000_max": 0.06481403998842197,
-            "nauc_map_at_1000_std": -0.002127813997161774,
-            "nauc_map_at_100_diff1": -0.27947218049898986,
-            "nauc_map_at_100_max": 0.07294318363029507,
-            "nauc_map_at_100_std": -0.03374119520674329,
-            "nauc_map_at_10_diff1": -0.28999308873919133,
-            "nauc_map_at_10_max": 0.05160832326140188,
+            "nauc_map_at_1000_diff1": -0.30660530667844693,
+            "nauc_map_at_1000_max": -0.0028096953436196454,
+            "nauc_map_at_1000_std": -0.00212684509251568,
+            "nauc_map_at_100_diff1": -0.27984389366501017,
+            "nauc_map_at_100_max": 0.009061580934529696,
+            "nauc_map_at_100_std": -0.03374029031888037,
+            "nauc_map_at_10_diff1": -0.2902609866914257,
+            "nauc_map_at_10_max": 0.011156540330714983,
             "nauc_map_at_10_std": -0.08882831762503994,
             "nauc_map_at_1_diff1": 0.24972628735650423,
-            "nauc_map_at_1_max": 0.04436445316921541,
+            "nauc_map_at_1_max": 0.07878869844045779,
             "nauc_map_at_1_std": -0.22104489503191824,
-            "nauc_map_at_20_diff1": -0.29026920494078134,
-            "nauc_map_at_20_max": 0.07182591908926299,
+            "nauc_map_at_20_diff1": -0.29070135277398945,
+            "nauc_map_at_20_max": 0.010382794620486961,
             "nauc_map_at_20_std": -0.07095213829913784,
             "nauc_map_at_3_diff1": -0.14883436753391752,
-            "nauc_map_at_3_max": 0.05147550207326568,
+            "nauc_map_at_3_max": 0.02270770366902631,
             "nauc_map_at_3_std": -0.11619818074979756,
-            "nauc_map_at_5_diff1": -0.2245041662418187,
-            "nauc_map_at_5_max": 0.05691598780155631,
+            "nauc_map_at_5_diff1": -0.22482351448896223,
+            "nauc_map_at_5_max": 0.02588361317830081,
             "nauc_map_at_5_std": -0.09219182058837157,
             "nauc_mrr_at_1000_diff1": null,
             "nauc_mrr_at_1000_max": null,
@@ -291,68 +291,68 @@
             "nauc_mrr_at_5_diff1": null,
             "nauc_mrr_at_5_max": null,
             "nauc_mrr_at_5_std": null,
-            "nauc_ndcg_at_1000_diff1": -0.3488605352692088,
-            "nauc_ndcg_at_1000_max": 0.05417572398363529,
-            "nauc_ndcg_at_1000_std": 0.045735272890686254,
-            "nauc_ndcg_at_100_diff1": -0.22470455576516485,
-            "nauc_ndcg_at_100_max": 0.07580255523108541,
-            "nauc_ndcg_at_100_std": -0.03429482520074303,
-            "nauc_ndcg_at_10_diff1": -0.477097231187033,
-            "nauc_ndcg_at_10_max": 0.03107634695495242,
+            "nauc_ndcg_at_1000_diff1": -0.3492256875331575,
+            "nauc_ndcg_at_1000_max": -0.016668948629775457,
+            "nauc_ndcg_at_1000_std": 0.045735876050701445,
+            "nauc_ndcg_at_100_diff1": -0.22528331233648932,
+            "nauc_ndcg_at_100_max": 0.020056171913496307,
+            "nauc_ndcg_at_100_std": -0.03429448936841829,
+            "nauc_ndcg_at_10_diff1": -0.4776622598008304,
+            "nauc_ndcg_at_10_max": -0.04144817199937437,
             "nauc_ndcg_at_10_std": 0.08553158841968203,
             "nauc_ndcg_at_1_diff1": null,
             "nauc_ndcg_at_1_max": null,
             "nauc_ndcg_at_1_std": null,
-            "nauc_ndcg_at_20_diff1": -0.40505487056335543,
-            "nauc_ndcg_at_20_max": 0.05265471832318506,
+            "nauc_ndcg_at_20_diff1": -0.40569970152070406,
+            "nauc_ndcg_at_20_max": -0.030056967718629198,
             "nauc_ndcg_at_20_std": 0.08329395449701402,
             "nauc_ndcg_at_3_diff1": -0.6600486434263803,
-            "nauc_ndcg_at_3_max": 0.04053615524180157,
+            "nauc_ndcg_at_3_max": -0.08703979992019233,
             "nauc_ndcg_at_3_std": 0.13595209257593371,
-            "nauc_ndcg_at_5_diff1": -0.5670342417973104,
-            "nauc_ndcg_at_5_max": 0.04140700244611174,
+            "nauc_ndcg_at_5_diff1": -0.5677191170570411,
+            "nauc_ndcg_at_5_max": -0.051263623328511726,
             "nauc_ndcg_at_5_std": 0.11754665940394998,
             "nauc_precision_at_1000_diff1": -0.21531952627685155,
-            "nauc_precision_at_1000_max": -0.04861090361583422,
+            "nauc_precision_at_1000_max": -0.0780721643759249,
             "nauc_precision_at_1000_std": 0.17003095488888245,
-            "nauc_precision_at_100_diff1": -0.26030299861833706,
-            "nauc_precision_at_100_max": 0.03134445708713708,
+            "nauc_precision_at_100_diff1": -0.26086429487194157,
+            "nauc_precision_at_100_max": -0.02125044974939926,
             "nauc_precision_at_100_std": 0.08134493216698098,
-            "nauc_precision_at_10_diff1": -0.4408010395929642,
-            "nauc_precision_at_10_max": 0.02966101658750455,
+            "nauc_precision_at_10_diff1": -0.44140500010863204,
+            "nauc_precision_at_10_max": -0.034976148499747076,
             "nauc_precision_at_10_std": 0.0764255856561166,
             "nauc_precision_at_1_diff1": null,
             "nauc_precision_at_1_max": null,
             "nauc_precision_at_1_std": null,
-            "nauc_precision_at_20_diff1": -0.35587590336185415,
-            "nauc_precision_at_20_max": 0.05843230329361628,
+            "nauc_precision_at_20_diff1": -0.35653616250077463,
+            "nauc_precision_at_20_max": -0.022643215643706124,
             "nauc_precision_at_20_std": 0.07573439130386994,
             "nauc_precision_at_3_diff1": -0.6507094768501036,
-            "nauc_precision_at_3_max": 0.03821197756709791,
+            "nauc_precision_at_3_max": -0.08424145859338003,
             "nauc_precision_at_3_std": 0.13333345746843658,
-            "nauc_precision_at_5_diff1": -0.5451253393111648,
-            "nauc_precision_at_5_max": 0.03989806995998538,
+            "nauc_precision_at_5_diff1": -0.5459627403516182,
+            "nauc_precision_at_5_max": -0.04290999641498163,
             "nauc_precision_at_5_std": 0.11333564980042542,
             "nauc_recall_at_1000_diff1": null,
             "nauc_recall_at_1000_max": null,
             "nauc_recall_at_1000_std": null,
-            "nauc_recall_at_100_diff1": -0.10473907849885522,
-            "nauc_recall_at_100_max": 0.07763109578818508,
+            "nauc_recall_at_100_diff1": -0.10534056523896512,
+            "nauc_recall_at_100_max": 0.03501195649235639,
             "nauc_recall_at_100_std": -0.06538749858536112,
-            "nauc_recall_at_10_diff1": -0.2130006791468074,
-            "nauc_recall_at_10_max": 0.05430726381941392,
+            "nauc_recall_at_10_diff1": -0.21357411304993232,
+            "nauc_recall_at_10_max": 0.02398007563666141,
             "nauc_recall_at_10_std": -0.10569357592600205,
             "nauc_recall_at_1_diff1": 0.24972628735650423,
-            "nauc_recall_at_1_max": 0.04436445316921541,
+            "nauc_recall_at_1_max": 0.07878869844045779,
             "nauc_recall_at_1_std": -0.22104489503191824,
-            "nauc_recall_at_20_diff1": -0.18587539581340096,
-            "nauc_recall_at_20_max": 0.08542381032015733,
+            "nauc_recall_at_20_diff1": -0.1864286024678164,
+            "nauc_recall_at_20_max": 0.031178653984817232,
             "nauc_recall_at_20_std": -0.08609480122819478,
             "nauc_recall_at_3_diff1": -0.13006888083893606,
-            "nauc_recall_at_3_max": 0.04617905393428449,
+            "nauc_recall_at_3_max": 0.019383132224575387,
             "nauc_recall_at_3_std": -0.12029056211294316,
-            "nauc_recall_at_5_diff1": -0.18408289754720042,
-            "nauc_recall_at_5_max": 0.05831772333514569,
+            "nauc_recall_at_5_diff1": -0.18462874347696806,
+            "nauc_recall_at_5_max": 0.03358718463953569,
             "nauc_recall_at_5_std": -0.09811503635318748,
             "ndcg_at_1": 1.0,
             "ndcg_at_10": 0.68438,
@@ -405,26 +405,26 @@
             "mrr_at_3": 1.0,
             "mrr_at_5": 1.0,
             "mteb_version": "2.21.0",
-            "nauc_map_at_1000_diff1": -0.34668216281081043,
-            "nauc_map_at_1000_max": -0.014147808043851405,
-            "nauc_map_at_1000_std": 0.46867769563250317,
-            "nauc_map_at_100_diff1": -0.23246990393164418,
-            "nauc_map_at_100_max": -0.03064595229312303,
-            "nauc_map_at_100_std": 0.2944328115688014,
-            "nauc_map_at_10_diff1": -0.017634923570675764,
-            "nauc_map_at_10_max": -0.07030913081569153,
+            "nauc_map_at_1000_diff1": -0.3466748695667919,
+            "nauc_map_at_1000_max": -0.06602021386289471,
+            "nauc_map_at_1000_std": 0.4686780435913505,
+            "nauc_map_at_100_diff1": -0.23246592003247477,
+            "nauc_map_at_100_max": -0.05786533298350443,
+            "nauc_map_at_100_std": 0.2944333254222356,
+            "nauc_map_at_10_diff1": -0.01766769209443787,
+            "nauc_map_at_10_max": 0.01662425583062413,
             "nauc_map_at_10_std": -0.20310264328314265,
             "nauc_map_at_1_diff1": 0.33748871614405934,
-            "nauc_map_at_1_max": -0.02813108852720766,
+            "nauc_map_at_1_max": 0.044909162722878745,
             "nauc_map_at_1_std": -0.40626289316703856,
-            "nauc_map_at_20_diff1": -0.07019698364979507,
-            "nauc_map_at_20_max": -0.06788974016713693,
+            "nauc_map_at_20_diff1": -0.0702109805957209,
+            "nauc_map_at_20_max": -0.013211894733465127,
             "nauc_map_at_20_std": -0.08635250527986776,
-            "nauc_map_at_3_diff1": 0.15107886225117026,
-            "nauc_map_at_3_max": -0.04288465628093434,
+            "nauc_map_at_3_diff1": 0.15097507697375423,
+            "nauc_map_at_3_max": 0.03818784442922138,
             "nauc_map_at_3_std": -0.3481925508466547,
-            "nauc_map_at_5_diff1": 0.06989436598086839,
-            "nauc_map_at_5_max": -0.055108302665684096,
+            "nauc_map_at_5_diff1": 0.06980268662028347,
+            "nauc_map_at_5_max": 0.02016135156329192,
             "nauc_map_at_5_std": -0.2967421139356778,
             "nauc_mrr_at_1000_diff1": null,
             "nauc_mrr_at_1000_max": null,
@@ -447,68 +447,68 @@
             "nauc_mrr_at_5_diff1": null,
             "nauc_mrr_at_5_max": null,
             "nauc_mrr_at_5_std": null,
-            "nauc_ndcg_at_1000_diff1": -0.41624555109843364,
-            "nauc_ndcg_at_1000_max": -0.004890748499403179,
-            "nauc_ndcg_at_1000_std": 0.4772177527918817,
-            "nauc_ndcg_at_100_diff1": -0.23814643269006355,
-            "nauc_ndcg_at_100_max": 0.0010155173133254564,
-            "nauc_ndcg_at_100_std": 0.2906804366969244,
-            "nauc_ndcg_at_10_diff1": -0.533411963114004,
-            "nauc_ndcg_at_10_max": -0.004943011227601837,
+            "nauc_ndcg_at_1000_diff1": -0.4162294230932042,
+            "nauc_ndcg_at_1000_max": -0.06215538491088406,
+            "nauc_ndcg_at_1000_std": 0.4772178426347044,
+            "nauc_ndcg_at_100_diff1": -0.23809596651185824,
+            "nauc_ndcg_at_100_max": -0.03194537552641834,
+            "nauc_ndcg_at_100_std": 0.2906805227408279,
+            "nauc_ndcg_at_10_diff1": -0.5334711615136456,
+            "nauc_ndcg_at_10_max": -0.0020786545226579817,
             "nauc_ndcg_at_10_std": 0.3774569217123403,
             "nauc_ndcg_at_1_diff1": null,
             "nauc_ndcg_at_1_max": null,
             "nauc_ndcg_at_1_std": null,
-            "nauc_ndcg_at_20_diff1": -0.44377365763912363,
-            "nauc_ndcg_at_20_max": 0.001762088675760577,
+            "nauc_ndcg_at_20_diff1": -0.4437631251502566,
+            "nauc_ndcg_at_20_max": -0.031231177455076014,
             "nauc_ndcg_at_20_std": 0.4130991024844574,
-            "nauc_ndcg_at_3_diff1": -0.726599688888509,
-            "nauc_ndcg_at_3_max": 0.02859748172207213,
+            "nauc_ndcg_at_3_diff1": -0.7273379726670222,
+            "nauc_ndcg_at_3_max": 0.006046945512110215,
             "nauc_ndcg_at_3_std": 0.30565575941152245,
-            "nauc_ndcg_at_5_diff1": -0.6401232803358068,
-            "nauc_ndcg_at_5_max": 0.005354055961325193,
+            "nauc_ndcg_at_5_diff1": -0.6404896483406627,
+            "nauc_ndcg_at_5_max": -0.017951839296109853,
             "nauc_ndcg_at_5_std": 0.3481495086877472,
             "nauc_precision_at_1000_diff1": -0.3880249341025205,
-            "nauc_precision_at_1000_max": 0.03662669641839722,
+            "nauc_precision_at_1000_max": -0.04902829086414168,
             "nauc_precision_at_1000_std": 0.5267883554911017,
-            "nauc_precision_at_100_diff1": -0.40389208711275676,
-            "nauc_precision_at_100_max": 0.038253694371980104,
+            "nauc_precision_at_100_diff1": -0.4038234507784032,
+            "nauc_precision_at_100_max": -0.05160718626450024,
             "nauc_precision_at_100_std": 0.6085834689608731,
-            "nauc_precision_at_10_diff1": -0.49976976365019193,
-            "nauc_precision_at_10_max": -0.008567475951651218,
+            "nauc_precision_at_10_diff1": -0.49976976365019254,
+            "nauc_precision_at_10_max": -0.0022455792680961964,
             "nauc_precision_at_10_std": 0.3843070115102614,
             "nauc_precision_at_1_diff1": null,
             "nauc_precision_at_1_max": null,
             "nauc_precision_at_1_std": null,
-            "nauc_precision_at_20_diff1": -0.40722959378663487,
-            "nauc_precision_at_20_max": 0.007627713227919489,
+            "nauc_precision_at_20_diff1": -0.4071671682209932,
+            "nauc_precision_at_20_max": -0.03580192357005322,
             "nauc_precision_at_20_std": 0.42986404518321963,
-            "nauc_precision_at_3_diff1": -0.708485500521119,
-            "nauc_precision_at_3_max": 0.0255586111875892,
+            "nauc_precision_at_3_diff1": -0.7093076105239234,
+            "nauc_precision_at_3_max": -0.005431412489875229,
             "nauc_precision_at_3_std": 0.3096370110430313,
-            "nauc_precision_at_5_diff1": -0.6186958143912717,
-            "nauc_precision_at_5_max": 0.001752460287016669,
+            "nauc_precision_at_5_diff1": -0.61903979796241,
+            "nauc_precision_at_5_max": -0.02599976900344492,
             "nauc_precision_at_5_std": 0.3566650159780995,
             "nauc_recall_at_1000_diff1": null,
             "nauc_recall_at_1000_max": null,
             "nauc_recall_at_1000_std": null,
-            "nauc_recall_at_100_diff1": -0.042533226539043935,
-            "nauc_recall_at_100_max": -0.015973581273468734,
+            "nauc_recall_at_100_diff1": -0.042459437594490744,
+            "nauc_recall_at_100_max": -0.05336644802413002,
             "nauc_recall_at_100_std": 0.14513576110445187,
             "nauc_recall_at_10_diff1": 0.061843254104618,
-            "nauc_recall_at_10_max": -0.06668017922938962,
+            "nauc_recall_at_10_max": 0.022012188384178285,
             "nauc_recall_at_10_std": -0.2545320599308331,
             "nauc_recall_at_1_diff1": 0.33748871614405934,
-            "nauc_recall_at_1_max": -0.02813108852720766,
+            "nauc_recall_at_1_max": 0.044909162722878745,
             "nauc_recall_at_1_std": -0.40626289316703856,
-            "nauc_recall_at_20_diff1": 0.05516859691929835,
-            "nauc_recall_at_20_max": -0.0576542535252272,
+            "nauc_recall_at_20_diff1": 0.05519986869358858,
+            "nauc_recall_at_20_max": -0.006374455148836075,
             "nauc_recall_at_20_std": -0.17425941998255615,
-            "nauc_recall_at_3_diff1": 0.1656973611520991,
-            "nauc_recall_at_3_max": -0.04229908581525453,
+            "nauc_recall_at_3_diff1": 0.16559359601100346,
+            "nauc_recall_at_3_max": 0.03741113830816123,
             "nauc_recall_at_3_std": -0.35444109202574064,
-            "nauc_recall_at_5_diff1": 0.10475299766223975,
-            "nauc_recall_at_5_max": -0.05128399259100251,
+            "nauc_recall_at_5_diff1": 0.10468080253603719,
+            "nauc_recall_at_5_max": 0.017080513860838822,
             "nauc_recall_at_5_std": -0.3134865171155313,
             "ndcg_at_1": 1.0,
             "ndcg_at_10": 0.75188,
@@ -561,26 +561,26 @@
             "mrr_at_3": 1.0,
             "mrr_at_5": 1.0,
             "mteb_version": "2.21.0",
-            "nauc_map_at_1000_diff1": -0.3459584027821315,
-            "nauc_map_at_1000_max": 0.002240266012098031,
-            "nauc_map_at_1000_std": 0.11180989634343157,
-            "nauc_map_at_100_diff1": -0.3385133550990697,
-            "nauc_map_at_100_max": -0.044096468597783975,
-            "nauc_map_at_100_std": 0.05808158927384358,
+            "nauc_map_at_1000_diff1": -0.3459579508173824,
+            "nauc_map_at_1000_max": -0.04978996128901983,
+            "nauc_map_at_1000_std": 0.11180887527818667,
+            "nauc_map_at_100_diff1": -0.3385131122051252,
+            "nauc_map_at_100_max": -0.08508689342435104,
+            "nauc_map_at_100_std": 0.058080953143544,
             "nauc_map_at_10_diff1": -0.3411659864964207,
-            "nauc_map_at_10_max": -0.05481421134918112,
+            "nauc_map_at_10_max": -0.11726463440046836,
             "nauc_map_at_10_std": -0.03129471392030425,
             "nauc_map_at_1_diff1": 0.08339661515546767,
-            "nauc_map_at_1_max": -0.12043622018612965,
+            "nauc_map_at_1_max": -0.11524013799329563,
             "nauc_map_at_1_std": -0.22155350932489112,
-            "nauc_map_at_20_diff1": -0.35916064876066994,
-            "nauc_map_at_20_max": -0.061958841418596776,
+            "nauc_map_at_20_diff1": -0.3591606487606701,
+            "nauc_map_at_20_max": -0.10315299204076839,
             "nauc_map_at_20_std": 0.03350316559200977,
             "nauc_map_at_3_diff1": -0.22973571313970262,
-            "nauc_map_at_3_max": -0.11645672062659865,
+            "nauc_map_at_3_max": -0.1344822377314527,
             "nauc_map_at_3_std": -0.11784418755975176,
             "nauc_map_at_5_diff1": -0.30328945559816733,
-            "nauc_map_at_5_max": -0.09412356934899256,
+            "nauc_map_at_5_max": -0.11888064885393486,
             "nauc_map_at_5_std": -0.08444366964004033,
             "nauc_mrr_at_1000_diff1": null,
             "nauc_mrr_at_1000_max": null,
@@ -603,68 +603,68 @@
             "nauc_mrr_at_5_diff1": null,
             "nauc_mrr_at_5_max": null,
             "nauc_mrr_at_5_std": null,
-            "nauc_ndcg_at_1000_diff1": -0.3803019202678134,
-            "nauc_ndcg_at_1000_max": 0.03985296875757049,
-            "nauc_ndcg_at_1000_std": 0.14600978304137952,
-            "nauc_ndcg_at_100_diff1": -0.2992985445385993,
-            "nauc_ndcg_at_100_max": -0.04182613222734348,
-            "nauc_ndcg_at_100_std": 0.013992645162876998,
+            "nauc_ndcg_at_1000_diff1": -0.38030181913549865,
+            "nauc_ndcg_at_1000_max": -0.030693798681738316,
+            "nauc_ndcg_at_1000_std": 0.14600917366468005,
+            "nauc_ndcg_at_100_diff1": -0.29929844312098247,
+            "nauc_ndcg_at_100_max": -0.09904482216835384,
+            "nauc_ndcg_at_100_std": 0.01399227172128409,
             "nauc_ndcg_at_10_diff1": -0.5056783608986034,
-            "nauc_ndcg_at_10_max": 0.03787165762781677,
+            "nauc_ndcg_at_10_max": -0.07557658766303663,
             "nauc_ndcg_at_10_std": 0.1554387091487298,
             "nauc_ndcg_at_1_diff1": null,
             "nauc_ndcg_at_1_max": null,
             "nauc_ndcg_at_1_std": null,
-            "nauc_ndcg_at_20_diff1": -0.4331557372974084,
-            "nauc_ndcg_at_20_max": 0.01697382465286896,
+            "nauc_ndcg_at_20_diff1": -0.43315573729740803,
+            "nauc_ndcg_at_20_max": -0.048350024477379246,
             "nauc_ndcg_at_20_std": 0.16496545740144006,
             "nauc_ndcg_at_3_diff1": -0.7668921432810086,
-            "nauc_ndcg_at_3_max": -0.031245373282396797,
+            "nauc_ndcg_at_3_max": -0.1266688249898702,
             "nauc_ndcg_at_3_std": 0.16389376917714546,
             "nauc_ndcg_at_5_diff1": -0.6391610785960832,
-            "nauc_ndcg_at_5_max": 0.0024436765212643226,
+            "nauc_ndcg_at_5_max": -0.08680009343431687,
             "nauc_ndcg_at_5_std": 0.15509801710735416,
             "nauc_precision_at_1000_diff1": -0.04021255877072952,
-            "nauc_precision_at_1000_max": 0.1733456196689033,
+            "nauc_precision_at_1000_max": 0.1578912923489832,
             "nauc_precision_at_1000_std": 0.21920853340217933,
-            "nauc_precision_at_100_diff1": -0.18116368379614803,
-            "nauc_precision_at_100_max": 0.08444510702143793,
+            "nauc_precision_at_100_diff1": -0.18116368379614733,
+            "nauc_precision_at_100_max": 0.036102428616712846,
             "nauc_precision_at_100_std": 0.1597185011148999,
-            "nauc_precision_at_10_diff1": -0.4617851971471919,
-            "nauc_precision_at_10_max": 0.04363662492887463,
+            "nauc_precision_at_10_diff1": -0.46178519714719163,
+            "nauc_precision_at_10_max": -0.06616555482750021,
             "nauc_precision_at_10_std": 0.15238572929252808,
             "nauc_precision_at_1_diff1": null,
             "nauc_precision_at_1_max": null,
             "nauc_precision_at_1_std": null,
             "nauc_precision_at_20_diff1": -0.38317777681531323,
-            "nauc_precision_at_20_max": 0.014060033568636781,
+            "nauc_precision_at_20_max": -0.03518113061394383,
             "nauc_precision_at_20_std": 0.16352974123299116,
             "nauc_precision_at_3_diff1": -0.7508165194229558,
-            "nauc_precision_at_3_max": -0.03684808454367177,
+            "nauc_precision_at_3_max": -0.11394714828725894,
             "nauc_precision_at_3_std": 0.16158866396407112,
             "nauc_precision_at_5_diff1": -0.6088963436251708,
-            "nauc_precision_at_5_max": 0.003965346915104591,
+            "nauc_precision_at_5_max": -0.07481977748166893,
             "nauc_precision_at_5_std": 0.15302454411965444,
             "nauc_recall_at_1000_diff1": null,
             "nauc_recall_at_1000_max": null,
             "nauc_recall_at_1000_std": null,
             "nauc_recall_at_100_diff1": -0.19128205558256184,
-            "nauc_recall_at_100_max": -0.03997275189642916,
+            "nauc_recall_at_100_max": -0.08964885984582112,
             "nauc_recall_at_100_std": -0.026496291343129752,
-            "nauc_recall_at_10_diff1": -0.28115496158825004,
-            "nauc_recall_at_10_max": -0.05772839208214799,
+            "nauc_recall_at_10_diff1": -0.2811549615882498,
+            "nauc_recall_at_10_max": -0.11779863813742072,
             "nauc_recall_at_10_std": -0.05800950816275074,
             "nauc_recall_at_1_diff1": 0.08339661515546767,
-            "nauc_recall_at_1_max": -0.12043622018612965,
+            "nauc_recall_at_1_max": -0.11524013799329563,
             "nauc_recall_at_1_std": -0.22155350932489112,
-            "nauc_recall_at_20_diff1": -0.2885469344555254,
-            "nauc_recall_at_20_max": -0.07353824301431425,
+            "nauc_recall_at_20_diff1": -0.28854693445552504,
+            "nauc_recall_at_20_max": -0.1053604626462759,
             "nauc_recall_at_20_std": -0.013919628913720157,
-            "nauc_recall_at_3_diff1": -0.21505758984328746,
-            "nauc_recall_at_3_max": -0.11843060227153768,
+            "nauc_recall_at_3_diff1": -0.2150575898432871,
+            "nauc_recall_at_3_max": -0.13211775211610113,
             "nauc_recall_at_3_std": -0.12403992079656084,
             "nauc_recall_at_5_diff1": -0.2642706458028187,
-            "nauc_recall_at_5_max": -0.09383504395152885,
+            "nauc_recall_at_5_max": -0.11618337672799929,
             "nauc_recall_at_5_std": -0.10214925094919192,
             "ndcg_at_1": 1.0,
             "ndcg_at_10": 0.70421,
@@ -689,7 +689,8 @@
             "recall_at_5": 0.08192
           }
         ]
-      }
+      },
+      "raw_result_sha256": "340d0bea12c2562c0e0b41e005bbbc112a586d7ef585113932d9c5b0e04d27ef"
     },
     {
       "task": "BeijingOpera",
@@ -784,7 +785,8 @@
             ]
           }
         ]
-      }
+      },
+      "raw_result_sha256": "fe00d9418a10c9c72d06b8e9c601abd3329ce3af94e0a3fe35b1af68ed644361"
     },
     {
       "task": "BirdCLEF",
@@ -879,7 +881,8 @@
             ]
           }
         ]
-      }
+      },
+      "raw_result_sha256": "0f74880aa6a0c1968dcd534e86b141e0954d837ecd5d58e4d3bec0cc6e06b881"
     },
     {
       "task": "CREMA_D",
@@ -974,7 +977,8 @@
             ]
           }
         ]
-      }
+      },
+      "raw_result_sha256": "dc628a2507c59041c28f0f44e25f3adcd5dc0e74e3f61f8d330119bba1089e5d"
     },
     {
       "task": "CommonLanguageAgeDetection",
@@ -1124,7 +1128,8 @@
             ]
           }
         ]
-      }
+      },
+      "raw_result_sha256": "a1b60b11ff8ed096a5d4aa7ca8e1131c37cc25ba05c751a087d4ba55dd6e35e2"
     },
     {
       "task": "GTZANGenre",
@@ -1219,7 +1224,8 @@
             ]
           }
         ]
-      }
+      },
+      "raw_result_sha256": "a41fe803722f639166cd5c61878e630cd0f97764edfe0a2c81537d2c96a8df08"
     },
     {
       "task": "IEMOCAPGender",
@@ -1314,7 +1320,8 @@
             ]
           }
         ]
-      }
+      },
+      "raw_result_sha256": "74a44e7d10fa5f5e435e4ac47deb493188b0501c9b3d2d29a754921af013eb76"
     },
     {
       "task": "MInDS14",
@@ -2234,14 +2241,15 @@
             ]
           }
         ]
-      }
+      },
+      "raw_result_sha256": "36361a313f59e6522f1085598112871cfd09dd859ed90457d85220ab1d149c97"
     },
     {
       "task": "MridinghamTonic",
       "task_type": "AudioClassification",
       "primary_metric": "accuracy",
-      "main_score": 0.32506577933881753,
-      "display_score_100": 32.50657793388175,
+      "main_score": 0.32363208758254514,
+      "display_score_100": 32.363208758254515,
       "dataset": {
         "path": "mteb/mridingham-tonic",
         "revision": "9304553355441e3d2bf2432691b6209ff9a9339c"
@@ -2255,21 +2263,21 @@
       "rich_scores": {
         "train": [
           {
-            "accuracy": 0.32506577933881753,
+            "accuracy": 0.32363208758254514,
             "ap": null,
             "ap_weighted": null,
-            "f1": 0.24370216344518064,
-            "f1_weighted": 0.2584177152428794,
+            "f1": 0.24214153797361287,
+            "f1_weighted": 0.25656361482363016,
             "hf_subset": "default",
             "languages": [
               "zxx-Zxxx"
             ],
-            "main_score": 0.32506577933881753,
+            "main_score": 0.32363208758254514,
             "mteb_version": "2.21.0",
-            "precision": 0.32218112715991465,
-            "precision_weighted": 0.34886124777395694,
-            "recall": 0.3183452284360094,
-            "recall_weighted": 0.32506577933881753,
+            "precision": 0.3223918377528778,
+            "precision_weighted": 0.34894443144492604,
+            "recall": 0.31717601209173785,
+            "recall_weighted": 0.32363208758254514,
             "scores_per_experiment": [
               {
                 "accuracy": 0.3159025787965616,
@@ -2294,15 +2302,15 @@
                 "recall_weighted": 0.3502865329512894
               },
               {
-                "accuracy": 0.32831541218637994,
+                "accuracy": 0.3211469534050179,
                 "ap": null,
                 "ap_weighted": null,
-                "f1": 0.2248491383869621,
-                "f1_weighted": 0.24699805264437122,
-                "precision": 0.35467682864258077,
-                "precision_weighted": 0.3873081237617785,
-                "recall": 0.3133653444192408,
-                "recall_weighted": 0.32831541218637994
+                "f1": 0.21704601102912346,
+                "f1_weighted": 0.237727550548125,
+                "precision": 0.3557303816073964,
+                "precision_weighted": 0.3877240421166238,
+                "recall": 0.30751926269788327,
+                "recall_weighted": 0.3211469534050179
               },
               {
                 "accuracy": 0.2910394265232975,
@@ -2329,7 +2337,8 @@
             ]
           }
         ]
-      }
+      },
+      "raw_result_sha256": "5b3afafb0d7ca35616584b36f26855dc24feec6b349f8b4273d8d0ee13e724f5"
     },
     {
       "task": "SIBFLEURS",
@@ -2898,16 +2907,16 @@
             "accuracy": 0.1774703557312253,
             "ap": null,
             "ap_weighted": null,
-            "f1": 0.13762110904968047,
-            "f1_weighted": 0.1435420552218971,
+            "f1": 0.13898165326736756,
+            "f1_weighted": 0.14519836992169008,
             "hf_subset": "bel_Cyrl",
             "languages": [
               "bel-Cyrl"
             ],
             "main_score": 0.1774703557312253,
             "mteb_version": "2.21.0",
-            "precision": 0.12557823129251702,
-            "precision_weighted": 0.13852813852813856,
+            "precision": 0.1303401360544218,
+            "precision_weighted": 0.1443252399774139,
             "recall": 0.19809523809523807,
             "recall_weighted": 0.1774703557312253,
             "scores_per_experiment": [
@@ -2926,10 +2935,10 @@
                 "accuracy": 0.17391304347826086,
                 "ap": null,
                 "ap_weighted": null,
-                "f1": 0.1717687074829932,
-                "f1_weighted": 0.11853002070393374,
-                "precision": 0.13945578231292516,
-                "precision_weighted": 0.10766045548654245,
+                "f1": 0.17857142857142858,
+                "f1_weighted": 0.12681159420289853,
+                "precision": 0.16326530612244897,
+                "precision_weighted": 0.13664596273291926,
                 "recall": 0.32142857142857145,
                 "recall_weighted": 0.17391304347826086
               },
@@ -9999,7 +10008,8 @@
             ]
           }
         ]
-      }
+      },
+      "raw_result_sha256": "ca8723bb541df665f3a1f773ac4bc7de1e997634d4b867bf8dc6e505f69492e9"
     },
     {
       "task": "VoxCelebSA",
@@ -10094,7 +10104,8 @@
             ]
           }
         ]
-      }
+      },
+      "raw_result_sha256": "c5f632bd227158ae7d2a6232e7e13b01750d9af8b4da351d20bf01031e43339e"
     },
     {
       "task": "VoxPopuliLanguageID",
@@ -10193,7 +10204,8 @@
             ]
           }
         ]
-      }
+      },
+      "raw_result_sha256": "8d38a521c1c3165f81e2c47f04b1e431d2b9866b3ff3f86ab135d7d36bbc532d"
     },
     {
       "task": "CREMA_DClustering",
@@ -10254,7 +10266,8 @@
             }
           }
         ]
-      }
+      },
+      "raw_result_sha256": "5bf7c0ee288b59c12fe79d0a4abe049fc7963b429f2619c94aaadaf2a41f661f"
     },
     {
       "task": "VehicleSoundClustering",
@@ -10315,7 +10328,8 @@
             }
           }
         ]
-      }
+      },
+      "raw_result_sha256": "bcab92999b4f6acaa0a7180711d75897edb55daabebd08cf77e0abe2df4ca091"
     },
     {
       "task": "VoxPopuliGenderClustering",
@@ -10380,14 +10394,15 @@
             }
           }
         ]
-      }
+      },
+      "raw_result_sha256": "c0049f4bca6ba5426678fcd0ad69e8c8ffc1f138514fec485057995e00360be2"
     },
     {
       "task": "CREMADPairClassification",
       "task_type": "AudioPairClassification",
       "primary_metric": "max_ap",
-      "main_score": 0.5408018633313862,
-      "display_score_100": 54.08018633313863,
+      "main_score": 0.5420123629581632,
+      "display_score_100": 54.20123629581632,
       "dataset": {
         "path": "mteb/CREMADPairClassification",
         "revision": "20701cbd51bf282719674c9f38ec91e698b62e48"
@@ -10402,17 +10417,17 @@
         "test": [
           {
             "cosine_accuracy": 0.5363489499192245,
-            "cosine_ap": 0.5408015973083137,
+            "cosine_ap": 0.5408018463331916,
             "cosine_f1": 0.6672661870503598,
             "cosine_precision": 0.500945179584121,
             "cosine_recall": 0.9989229940764675,
             "dot_accuracy": 0.5363489499192245,
-            "dot_ap": 0.5408015227909019,
+            "dot_ap": 0.5408017434815562,
             "dot_f1": 0.6672661870503598,
             "dot_precision": 0.500945179584121,
             "dot_recall": 0.9989229940764675,
             "euclidean_accuracy": 0.5363489499192245,
-            "euclidean_ap": 0.5408018633313862,
+            "euclidean_ap": 0.540801808734759,
             "euclidean_f1": 0.6672661870503598,
             "euclidean_precision": 0.500945179584121,
             "euclidean_recall": 0.9989229940764675,
@@ -10420,33 +10435,34 @@
             "languages": [
               "eng-Latn"
             ],
-            "main_score": 0.5408018633313862,
-            "manhattan_accuracy": 0.5329833064081853,
-            "manhattan_ap": 0.5387179161276681,
-            "manhattan_f1": 0.6672072072072072,
-            "manhattan_precision": 0.5013539128080151,
-            "manhattan_recall": 0.9970382337102854,
+            "main_score": 0.5420123629581632,
+            "manhattan_accuracy": 0.5358104469574583,
+            "manhattan_ap": 0.5420123629581632,
+            "manhattan_f1": 0.6674470959027465,
+            "manhattan_precision": 0.5014206467325125,
+            "manhattan_recall": 0.9978459881529348,
             "max_accuracy": 0.5363489499192245,
-            "max_ap": 0.5408018633313862,
-            "max_f1": 0.6672661870503598,
-            "max_precision": 0.5013539128080151,
+            "max_ap": 0.5420123629581632,
+            "max_f1": 0.6674470959027465,
+            "max_precision": 0.5014206467325125,
             "max_recall": 0.9989229940764675,
             "mteb_version": "2.21.0",
             "similarity_accuracy": 0.5363489499192245,
-            "similarity_ap": 0.540801763292346,
+            "similarity_ap": 0.5408016814169725,
             "similarity_f1": 0.6672661870503598,
             "similarity_precision": 0.500945179584121,
             "similarity_recall": 0.9989229940764675
           }
         ]
-      }
+      },
+      "raw_result_sha256": "8fc37576f94f36ac826306ba427b0bb621d0642fd7cd11908acacc6e58878c49"
     },
     {
       "task": "NMSQAPairClassification",
       "task_type": "AudioPairClassification",
       "primary_metric": "max_ap",
-      "main_score": 0.6343546123003914,
-      "display_score_100": 63.43546123003914,
+      "main_score": 0.6328340215485784,
+      "display_score_100": 63.28340215485784,
       "dataset": {
         "path": "mteb/NMSQAPairClassification",
         "revision": "d33eea302d8a64c0a2ee094cf39d77c0814fe87a"
@@ -10479,16 +10495,16 @@
             "languages": [
               "eng-Latn"
             ],
-            "main_score": 0.6343546123003914,
-            "manhattan_accuracy": 0.6257309941520468,
-            "manhattan_ap": 0.6343546123003914,
-            "manhattan_f1": 0.691358024691358,
-            "manhattan_precision": 0.5316455696202531,
-            "manhattan_recall": 0.9882352941176471,
+            "main_score": 0.6328340215485784,
+            "manhattan_accuracy": 0.6198830409356725,
+            "manhattan_ap": 0.6328340215485784,
+            "manhattan_f1": 0.6882591093117408,
+            "manhattan_precision": 0.5246913580246914,
+            "manhattan_recall": 1.0,
             "max_accuracy": 0.631578947368421,
-            "max_ap": 0.6343546123003914,
+            "max_ap": 0.6328340215485784,
             "max_f1": 0.6938775510204082,
-            "max_precision": 0.5316455696202531,
+            "max_precision": 0.53125,
             "max_recall": 1.0,
             "mteb_version": "2.21.0",
             "similarity_accuracy": 0.631578947368421,
@@ -10498,14 +10514,15 @@
             "similarity_recall": 1.0
           }
         ]
-      }
+      },
+      "raw_result_sha256": "f7de28794644458dd9454b128a322bb03f3277e16cbd5950cdbddfee4de1de0f"
     },
     {
       "task": "VoxPopuliAccentPairClassification",
       "task_type": "AudioPairClassification",
       "primary_metric": "max_ap",
-      "main_score": 0.5405044619214763,
-      "display_score_100": 54.050446192147625,
+      "main_score": 0.5402200187526489,
+      "display_score_100": 54.02200187526489,
       "dataset": {
         "path": "mteb/VoxPopuliAccentPairClassificationFiltered",
         "revision": "ac4587795abbac156982e41775e1428f5933a32e"
@@ -10520,17 +10537,17 @@
         "test": [
           {
             "cosine_accuracy": 0.5296575896468051,
-            "cosine_ap": 0.5402197772626384,
+            "cosine_ap": 0.5402197808405574,
             "cosine_f1": 0.6663070837828119,
             "cosine_precision": 0.4996629364972361,
             "cosine_recall": 0.9997302400863232,
             "dot_accuracy": 0.5296575896468051,
-            "dot_ap": 0.5402199966374358,
+            "dot_ap": 0.5402200187526489,
             "dot_f1": 0.6663070837828119,
             "dot_precision": 0.4996629364972361,
             "dot_recall": 0.9997302400863232,
             "euclidean_accuracy": 0.5296575896468051,
-            "euclidean_ap": 0.5402198393477347,
+            "euclidean_ap": 0.5402196904542975,
             "euclidean_f1": 0.6663070837828119,
             "euclidean_precision": 0.4996629364972361,
             "euclidean_recall": 0.9997302400863232,
@@ -10538,26 +10555,27 @@
             "languages": [
               "eng-Latn"
             ],
-            "main_score": 0.5405044619214763,
-            "manhattan_accuracy": 0.5318145052574818,
-            "manhattan_ap": 0.5405044619214763,
-            "manhattan_f1": 0.6669068636281752,
-            "manhattan_precision": 0.5006085192697769,
-            "manhattan_recall": 0.9986512004316158,
-            "max_accuracy": 0.5318145052574818,
-            "max_ap": 0.5405044619214763,
-            "max_f1": 0.6669068636281752,
-            "max_precision": 0.5006085192697769,
+            "main_score": 0.5402200187526489,
+            "manhattan_accuracy": 0.5293879751954704,
+            "manhattan_ap": 0.5393917252850706,
+            "manhattan_f1": 0.6663669873235638,
+            "manhattan_precision": 0.49973031283710895,
+            "manhattan_recall": 0.9997302400863232,
+            "max_accuracy": 0.5296575896468051,
+            "max_ap": 0.5402200187526489,
+            "max_f1": 0.6663669873235638,
+            "max_precision": 0.49973031283710895,
             "max_recall": 0.9997302400863232,
             "mteb_version": "2.21.0",
             "similarity_accuracy": 0.5296575896468051,
-            "similarity_ap": 0.5402199966374358,
+            "similarity_ap": 0.5402199122248765,
             "similarity_f1": 0.6663070837828119,
             "similarity_precision": 0.4996629364972361,
             "similarity_recall": 0.9997302400863232
           }
         ]
-      }
+      },
+      "raw_result_sha256": "1d453888caf5b4c96dd20ed8cc5964525753b785893a991cbd5750e9300095bc"
     },
     {
       "task": "GTZANAudioReranking",
@@ -10734,17 +10752,1105 @@
             "recall_at_5": 0.72667
           }
         ]
-      }
+      },
+      "raw_result_sha256": "f63996228ff215e374ab556c7fc0b66c476b32f5d5611fc98a1952a8286de21b"
+    }
+  ],
+  "parameters": 144554112,
+  "revision_semantics": "Native artifact fingerprint identifying the evaluated weights and inference files; the release commit is the repository revision containing this report.",
+  "slots": [
+    {
+      "hf_subset": "en",
+      "main_score": 0.61555,
+      "metric": "ndcg_at_10",
+      "split": "test",
+      "task": "JamAltArtistA2ARetrieval",
+      "type": "Any2AnyRetrieval"
+    },
+    {
+      "hf_subset": "fr",
+      "main_score": 0.68438,
+      "metric": "ndcg_at_10",
+      "split": "test",
+      "task": "JamAltArtistA2ARetrieval",
+      "type": "Any2AnyRetrieval"
+    },
+    {
+      "hf_subset": "de",
+      "main_score": 0.75188,
+      "metric": "ndcg_at_10",
+      "split": "test",
+      "task": "JamAltArtistA2ARetrieval",
+      "type": "Any2AnyRetrieval"
+    },
+    {
+      "hf_subset": "es",
+      "main_score": 0.70421,
+      "metric": "ndcg_at_10",
+      "split": "test",
+      "task": "JamAltArtistA2ARetrieval",
+      "type": "Any2AnyRetrieval"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.686790780141844,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "BeijingOpera",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.129,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "BirdCLEF",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.29763670140167686,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "CREMA_D",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.1604,
+      "metric": "accuracy",
+      "split": "test",
+      "task": "CommonLanguageAgeDetection",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.503,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "GTZANGenre",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.5845186270364481,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "IEMOCAPGender",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "en-GB",
+      "main_score": 0.3950861700612448,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "MInDS14",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "fr-FR",
+      "main_score": 0.20581516095534785,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "MInDS14",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "it-IT",
+      "main_score": 0.1996916752312436,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "MInDS14",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "es-ES",
+      "main_score": 0.1872291184515043,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "MInDS14",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "pt-PT",
+      "main_score": 0.23677685950413224,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "MInDS14",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "de-DE",
+      "main_score": 0.27014527522324405,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "MInDS14",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "nl-NL",
+      "main_score": 0.25073399882560193,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "MInDS14",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ru-RU",
+      "main_score": 0.20219799238490826,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "MInDS14",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "pl-PL",
+      "main_score": 0.19213021491782553,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "MInDS14",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "cs-CZ",
+      "main_score": 0.23163996948893972,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "MInDS14",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ko-KR",
+      "main_score": 0.23986611593790058,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "MInDS14",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "zh-CN",
+      "main_score": 0.18722772277227723,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "MInDS14",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.32363208758254514,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "MridinghamTonic",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "afr_Latn",
+      "main_score": 0.18735177865612646,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "amh_Ethi",
+      "main_score": 0.18695652173913044,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "arb_Arab",
+      "main_score": 0.13399209486166005,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "asm_Beng",
+      "main_score": 0.13438735177865613,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ast_Latn",
+      "main_score": 0.28616600790513835,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "azj_Latn",
+      "main_score": 0.1691699604743083,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "bel_Cyrl",
+      "main_score": 0.1774703557312253,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "bul_Cyrl",
+      "main_score": 0.20553359683794464,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ben_Beng",
+      "main_score": 0.19604743083003956,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "bos_Latn",
+      "main_score": 0.20553359683794467,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "cat_Latn",
+      "main_score": 0.17984189723320157,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ceb_Latn",
+      "main_score": 0.2952569169960474,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ckb_Arab",
+      "main_score": 0.16126482213438736,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "zho_Hans",
+      "main_score": 0.1158102766798419,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ces_Latn",
+      "main_score": 0.1616600790513834,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "cym_Latn",
+      "main_score": 0.18023715415019764,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "dan_Latn",
+      "main_score": 0.1616600790513834,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "deu_Latn",
+      "main_score": 0.2316205533596838,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ell_Grek",
+      "main_score": 0.11660079051383399,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "eng_Latn",
+      "main_score": 0.20632411067193676,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "spa_Latn",
+      "main_score": 0.12450592885375496,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "est_Latn",
+      "main_score": 0.1782608695652174,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "pes_Arab",
+      "main_score": 0.15296442687747036,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "fin_Latn",
+      "main_score": 0.19644268774703558,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "tgl_Latn",
+      "main_score": 0.19525691699604744,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "fra_Latn",
+      "main_score": 0.22213438735177862,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "gle_Latn",
+      "main_score": 0.1972332015810277,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "glg_Latn",
+      "main_score": 0.133201581027668,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "guj_Gujr",
+      "main_score": 0.19802371541501979,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "hau_Latn",
+      "main_score": 0.2150197628458498,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "heb_Hebr",
+      "main_score": 0.17865612648221346,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "hin_Deva",
+      "main_score": 0.14347826086956522,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "hrv_Latn",
+      "main_score": 0.12450592885375493,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "hun_Latn",
+      "main_score": 0.12529644268774703,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "hye_Armn",
+      "main_score": 0.2316205533596838,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ind_Latn",
+      "main_score": 0.1960474308300395,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ibo_Latn",
+      "main_score": 0.25849802371541497,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "isl_Latn",
+      "main_score": 0.23320158102766797,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ita_Latn",
+      "main_score": 0.20592885375494072,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "jpn_Jpan",
+      "main_score": 0.15138339920948618,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "jav_Latn",
+      "main_score": 0.20553359683794467,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "kat_Geor",
+      "main_score": 0.11541501976284585,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "kam_Latn",
+      "main_score": 0.10671936758893281,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "kea_Latn",
+      "main_score": 0.258498023715415,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "kaz_Cyrl",
+      "main_score": 0.13359683794466404,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "khm_Khmr",
+      "main_score": 0.2316205533596838,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "kan_Knda",
+      "main_score": 0.16047430830039527,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "kor_Hang",
+      "main_score": 0.22411067193675888,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "kir_Cyrl",
+      "main_score": 0.20513833992094863,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ltz_Latn",
+      "main_score": 0.23241106719367588,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "lug_Latn",
+      "main_score": 0.18814229249011855,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "lin_Latn",
+      "main_score": 0.1869565217391304,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "lao_Laoo",
+      "main_score": 0.16047430830039527,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "lit_Latn",
+      "main_score": 0.21422924901185772,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "luo_Latn",
+      "main_score": 0.17826086956521742,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "lvs_Latn",
+      "main_score": 0.19683794466403165,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "mri_Latn",
+      "main_score": 0.241501976284585,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "mkd_Cyrl",
+      "main_score": 0.18695652173913044,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "mal_Mlym",
+      "main_score": 0.1964426877470356,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "khk_Cyrl",
+      "main_score": 0.3027667984189723,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "mar_Deva",
+      "main_score": 0.1691699604743083,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "zsm_Latn",
+      "main_score": 0.16798418972332016,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "mlt_Latn",
+      "main_score": 0.16086956521739132,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "mya_Mymr",
+      "main_score": 0.14308300395256918,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "nob_Latn",
+      "main_score": 0.241501976284585,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "npi_Deva",
+      "main_score": 0.12569169960474308,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "nld_Latn",
+      "main_score": 0.19604743083003956,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "nso_Latn",
+      "main_score": 0.17786561264822134,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "nya_Latn",
+      "main_score": 0.11620553359683794,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "oci_Latn",
+      "main_score": 0.24980237154150195,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ory_Orya",
+      "main_score": 0.07154150197628459,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "pan_Guru",
+      "main_score": 0.13438735177865613,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "pol_Latn",
+      "main_score": 0.18537549407114623,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "pbt_Arab",
+      "main_score": 0.15217391304347827,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "por_Latn",
+      "main_score": 0.20474308300395255,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ron_Latn",
+      "main_score": 0.13280632411067192,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "rus_Cyrl",
+      "main_score": 0.23359683794466402,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "snd_Arab",
+      "main_score": 0.13399209486166008,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "slk_Latn",
+      "main_score": 0.09762845849802373,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "slv_Latn",
+      "main_score": 0.16126482213438736,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "sna_Latn",
+      "main_score": 0.22371541501976283,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "som_Latn",
+      "main_score": 0.13280632411067192,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "srp_Cyrl",
+      "main_score": 0.15217391304347824,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "swe_Latn",
+      "main_score": 0.18656126482213437,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "swh_Latn",
+      "main_score": 0.25731225296442684,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "tam_Taml",
+      "main_score": 0.20553359683794464,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "tel_Telu",
+      "main_score": 0.2043478260869565,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "tgk_Cyrl",
+      "main_score": 0.18735177865612648,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "tha_Thai",
+      "main_score": 0.13399209486166008,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "tur_Latn",
+      "main_score": 0.22371541501976283,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "ukr_Cyrl",
+      "main_score": 0.24110671936758893,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "umb_Latn",
+      "main_score": 0.17865612648221343,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "urd_Arab",
+      "main_score": 0.17944664031620555,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "uzn_Latn",
+      "main_score": 0.16956521739130437,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "vie_Latn",
+      "main_score": 0.25770750988142294,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "wol_Latn",
+      "main_score": 0.12569169960474308,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "xho_Latn",
+      "main_score": 0.16205533596837945,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "yor_Latn",
+      "main_score": 0.18774703557312256,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "zho_Hant",
+      "main_score": 0.16126482213438736,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "zul_Latn",
+      "main_score": 0.2146245059288537,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "fuv_Latn",
+      "main_score": 0.16126482213438736,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "gaz_Latn",
+      "main_score": 0.18774703557312256,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "SIBFLEURS",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.33169138217538546,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "VoxCelebSA",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.9179999999999999,
+      "metric": "accuracy",
+      "split": "train",
+      "task": "VoxPopuliLanguageID",
+      "type": "AudioClassification"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.023601715918443712,
+      "metric": "v_measure",
+      "split": "train",
+      "task": "CREMA_DClustering",
+      "type": "AudioClustering"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.13800701178563807,
+      "metric": "v_measure",
+      "split": "test",
+      "task": "VehicleSoundClustering",
+      "type": "AudioClustering"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.0010452967111713475,
+      "metric": "v_measure",
+      "split": "train",
+      "task": "VoxPopuliGenderClustering",
+      "type": "AudioClustering"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.5420123629581632,
+      "metric": "max_ap",
+      "split": "test",
+      "task": "CREMADPairClassification",
+      "type": "AudioPairClassification"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.6328340215485784,
+      "metric": "max_ap",
+      "split": "test",
+      "task": "NMSQAPairClassification",
+      "type": "AudioPairClassification"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.5402200187526489,
+      "metric": "max_ap",
+      "split": "test",
+      "task": "VoxPopuliAccentPairClassification",
+      "type": "AudioPairClassification"
+    },
+    {
+      "hf_subset": "default",
+      "main_score": 0.63794,
+      "metric": "map_at_1000",
+      "split": "test",
+      "task": "GTZANAudioReranking",
+      "type": "AudioReranking"
     }
   ],
-  "component_applicability": {
-    "applies_to_revision": "self",
-    "applies_to_native_artifact_sha256": "fb9cd39c9980a085b75456c715c2295b997784fe733314be68c13635f6384edf",
-    "evaluated_revision": "70c3412bb0ddfb19386d673d4a102c753a574f1b",
-    "evaluated_native_artifact_sha256": "b596f1dbab89369555e0a72a0d6854d780e305686750949b308c8f167993bdeb",
-    "basis": "Exact component tensor and inference-file identity; original measured identity is retained.",
-    "report": "benchmarks/component-equivalence.json",
-    "new_full_panel_inference": false,
-    "component": "audio"
+  "comparison_to_previous_release": {
+    "BeijingOpera": 0.0,
+    "BirdCLEF": 0.0,
+    "CREMADPairClassification": 0.12104996267769952,
+    "CREMA_D": 0.0,
+    "CREMA_DClustering": 0.0,
+    "CommonLanguageAgeDetection": 0.0,
+    "GTZANAudioReranking": 0.0,
+    "GTZANGenre": 0.0,
+    "IEMOCAPGender": 0.0,
+    "JamAltArtistA2ARetrieval": 0.0,
+    "MInDS14": 0.0,
+    "MridinghamTonic": -0.14336917562723928,
+    "NMSQAPairClassification": -0.1520590751812989,
+    "SIBFLEURS": 0.0,
+    "VehicleSoundClustering": 0.0,
+    "VoxCelebSA": 0.0,
+    "VoxPopuliAccentPairClassification": -0.02844431688273641,
+    "VoxPopuliGenderClustering": 0.0,
+    "VoxPopuliLanguageID": 0.0
   }
 }

benchmarks/net-progress.json

@@ -0,0 +1,540 @@
+{
+  "scope": "Product-weighted progress comparison, not an official benchmark aggregate or SOTA claim.",
+  "baseline_model": "llm-semantic-router/Vela-1.0-Omni-Nano",
+  "baseline_revision": "3a1efc48995e2b71622d65c6cf71cfc00bbf3aa3",
+  "baseline_native_artifact_sha256": "fb9cd39c9980a085b75456c715c2295b997784fe733314be68c13635f6384edf",
+  "candidate_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
+  "score_scale": "0-100; changes in percentage points",
+  "components": [
+    {
+      "baseline_100": 59.37287180704907,
+      "candidate_100": 67.57407300861536,
+      "component": "common14",
+      "delta_pp": 8.201201201566288,
+      "weight": 0.3
+    },
+    {
+      "baseline_100": 51.97553164396411,
+      "candidate_100": 60.781849756549235,
+      "component": "English41_MeanTaskType",
+      "delta_pp": 8.806318112585124,
+      "weight": 0.3
+    },
+    {
+      "baseline_100": 54.865785925824184,
+      "candidate_100": 53.65746394890041,
+      "component": "image3",
+      "delta_pp": -1.208321976923776,
+      "weight": 0.2
+    },
+    {
+      "baseline_100": 46.9743843608,
+      "candidate_100": 46.96781408655678,
+      "component": "audio19_MeanTaskType",
+      "delta_pp": -0.00657027424322,
+      "weight": 0.2
+    }
+  ],
+  "weighted_net_delta_pp": 4.859277344012025,
+  "common_definition": "Equal mean of three families: two text accuracies, all six image retrieval recalls, all six audio retrieval recalls.",
+  "image3_definition": "Equal mean of OxfordPets classification accuracy, TinyImageNet clustering NMI and OxfordPets zero-shot accuracy.",
+  "text_audio_definition": "Full English41 and full audio19 mean task-type scores. English uses the explicit exact text-component bridge; audio is freshly evaluated.",
+  "coverage": {
+    "common_metrics": 14,
+    "standard_tasks": 11,
+    "standard_split_rows": 12,
+    "English_tasks": 41,
+    "audio_tasks": 19,
+    "audio_subset_split_slots": 134
+  },
+  "common_metric_delta_pp": {
+    "audio.audio_to_text.recall@1": 8.157793948678668,
+    "audio.audio_to_text.recall@10": 15.51129835312141,
+    "audio.audio_to_text.recall@5": 14.362313289927231,
+    "audio.text_to_audio.recall@1": 11.417624521072797,
+    "audio.text_to_audio.recall@10": 15.478927203065133,
+    "audio.text_to_audio.recall@5": 14.444444444444445,
+    "image.image_to_text.recall@1": -4.738760631834751,
+    "image.image_to_text.recall@10": -1.2150668286755772,
+    "image.image_to_text.recall@5": -1.701093560145808,
+    "image.text_to_image.recall@1": 2.7460510328068044,
+    "image.text_to_image.recall@10": 0.7776427703523694,
+    "image.text_to_image.recall@5": 1.0206561360874848,
+    "text.banking77.accuracy": 11.168831168831169,
+    "text.massive-en.accuracy": 12.617765814266487
+  },
+  "standard_task_deltas": [
+    {
+      "baseline": 0.7169618925026362,
+      "candidate": 0.8053174565838819,
+      "hf_subset": "default",
+      "main_metric": "cosine_spearman",
+      "split": "test",
+      "status": "complete",
+      "task": "SICK-R",
+      "delta_pp": 8.835556408124567
+    },
+    {
+      "baseline": 0.7792400226361769,
+      "candidate": 0.870781967302773,
+      "hf_subset": "default",
+      "main_metric": "cosine_spearman",
+      "split": "test",
+      "status": "complete",
+      "task": "STSBenchmark",
+      "delta_pp": 9.154194466659614
+    },
+    {
+      "baseline": 0.37514,
+      "candidate": 0.59246,
+      "hf_subset": "default",
+      "main_metric": "ndcg_at_10",
+      "split": "test",
+      "status": "complete",
+      "task": "ArguAna",
+      "delta_pp": 21.732000000000003
+    },
+    {
+      "baseline": 0.44884717457494344,
+      "candidate": 0.5087260708286914,
+      "hf_subset": "default",
+      "main_metric": "v_measure",
+      "split": "test",
+      "status": "complete",
+      "task": "TwentyNewsgroupsClustering.v2",
+      "delta_pp": 5.987889625374793
+    },
+    {
+      "baseline": 0.8758836689038031,
+      "candidate": 0.9476510067114093,
+      "hf_subset": "en",
+      "main_metric": "accuracy",
+      "split": "validation",
+      "status": "complete",
+      "task": "MTOPDomainClassification",
+      "delta_pp": 7.176733780760625
+    },
+    {
+      "baseline": 0.8671910624715002,
+      "candidate": 0.9491792065663475,
+      "hf_subset": "en",
+      "main_metric": "accuracy",
+      "split": "test",
+      "status": "complete",
+      "task": "MTOPDomainClassification",
+      "delta_pp": 8.198814409484722
+    },
+    {
+      "baseline": 0.9126737530662306,
+      "candidate": 0.9126737530662306,
+      "hf_subset": "default",
+      "main_metric": "accuracy",
+      "split": "test",
+      "status": "complete",
+      "task": "OxfordPets",
+      "delta_pp": 0.0
+    },
+    {
+      "baseline": 0.6169193395626786,
+      "candidate": 0.6169193395626786,
+      "hf_subset": "default",
+      "main_metric": "nmi",
+      "split": "valid",
+      "status": "complete",
+      "task": "TinyImageNetClustering",
+      "delta_pp": 0.0
+    },
+    {
+      "baseline": 0.1163804851458163,
+      "candidate": 0.08013082583810302,
+      "hf_subset": "default",
+      "main_metric": "accuracy",
+      "split": "test",
+      "status": "complete",
+      "task": "OxfordPetsZeroShot",
+      "delta_pp": -3.6249659307713276
+    },
+    {
+      "baseline": 0.29763670140167686,
+      "candidate": 0.29763670140167686,
+      "hf_subset": "default",
+      "main_metric": "accuracy",
+      "split": "train",
+      "status": "complete",
+      "task": "CREMA_D",
+      "delta_pp": 0.0
+    },
+    {
+      "baseline": 0.023601715918443712,
+      "candidate": 0.023601715918443712,
+      "hf_subset": "default",
+      "main_metric": "v_measure",
+      "split": "train",
+      "status": "complete",
+      "task": "CREMA_DClustering",
+      "delta_pp": 0.0
+    },
+    {
+      "baseline": 0.14972999509081983,
+      "candidate": 0.1411389297987236,
+      "hf_subset": "default",
+      "main_metric": "accuracy",
+      "split": "test",
+      "status": "complete",
+      "task": "SpeechCommandsZeroshotv0.02",
+      "delta_pp": -0.8591065292096217
+    }
+  ],
+  "English_task_deltas": [
+    {
+      "candidate_100": 59.245999999999995,
+      "current_100": 37.513999999999996,
+      "delta_pp": 21.732,
+      "primary_metric": "ndcg_at_10",
+      "task": "ArguAna",
+      "task_type": "Retrieval"
+    },
+    {
+      "candidate_100": 64.88847259326147,
+      "current_100": 65.25395409281045,
+      "delta_pp": -0.36548149954897724,
+      "primary_metric": "v_measure",
+      "task": "ArXivHierarchicalClusteringP2P",
+      "task_type": "Clustering"
+    },
+    {
+      "candidate_100": 57.44315313869382,
+      "current_100": 55.989532291560415,
+      "delta_pp": 1.4536208471334078,
+      "primary_metric": "v_measure",
+      "task": "ArXivHierarchicalClusteringS2S",
+      "task_type": "Clustering"
+    },
+    {
+      "candidate_100": 62.329,
+      "current_100": 60.514,
+      "delta_pp": 1.8149999999999977,
+      "primary_metric": "map_at_1000",
+      "task": "AskUbuntuDupQuestions",
+      "task_type": "Reranking"
+    },
+    {
+      "candidate_100": 86.98752423842483,
+      "current_100": 62.24539420714732,
+      "delta_pp": 24.742130031277505,
+      "primary_metric": "cosine_spearman",
+      "task": "BIOSSES",
+      "task_type": "STS"
+    },
+    {
+      "candidate_100": 82.14285714285714,
+      "current_100": 69.57467532467533,
+      "delta_pp": 12.568181818181813,
+      "primary_metric": "accuracy",
+      "task": "Banking77Classification",
+      "task_type": "Classification"
+    },
+    {
+      "candidate_100": 41.313796273674654,
+      "current_100": 39.82923119776807,
+      "delta_pp": 1.4845650759065876,
+      "primary_metric": "v_measure",
+      "task": "BiorxivClusteringP2P.v2",
+      "task_type": "Clustering"
+    },
+    {
+      "candidate_100": 56.979,
+      "current_100": 42.976,
+      "delta_pp": 14.003,
+      "primary_metric": "ndcg_at_10",
+      "task": "CQADupstackGamingRetrieval",
+      "task_type": "Retrieval"
+    },
+    {
+      "candidate_100": 39.586,
+      "current_100": 30.362000000000002,
+      "delta_pp": 9.223999999999997,
+      "primary_metric": "ndcg_at_10",
+      "task": "CQADupstackUnixRetrieval",
+      "task_type": "Retrieval"
+    },
+    {
+      "candidate_100": 31.846000000000004,
+      "current_100": 16.073999999999998,
+      "delta_pp": 15.772000000000006,
+      "primary_metric": "ndcg_at_10",
+      "task": "ClimateFEVERHardNegatives",
+      "task_type": "Retrieval"
+    },
+    {
+      "candidate_100": 87.576,
+      "current_100": 24.055,
+      "delta_pp": 63.520999999999994,
+      "primary_metric": "ndcg_at_10",
+      "task": "FEVERHardNegatives",
+      "task_type": "Retrieval"
+    },
+    {
+      "candidate_100": 39.143,
+      "current_100": 22.588,
+      "delta_pp": 16.555,
+      "primary_metric": "ndcg_at_10",
+      "task": "FiQA2018",
+      "task_type": "Retrieval"
+    },
+    {
+      "candidate_100": 66.349,
+      "current_100": 26.987,
+      "delta_pp": 39.36200000000001,
+      "primary_metric": "ndcg_at_10",
+      "task": "HotpotQAHardNegatives",
+      "task_type": "Retrieval"
+    },
+    {
+      "candidate_100": 91.94760000000001,
+      "current_100": 62.1364,
+      "delta_pp": 29.811200000000007,
+      "primary_metric": "accuracy",
+      "task": "ImdbClassification",
+      "task_type": "Classification"
+    },
+    {
+      "candidate_100": 94.91792065663475,
+      "current_100": 86.71910624715002,
+      "delta_pp": 8.198814409484726,
+      "primary_metric": "accuracy",
+      "task": "MTOPDomainClassification",
+      "task_type": "Classification"
+    },
+    {
+      "candidate_100": 70.96839273705447,
+      "current_100": 60.39340954942838,
+      "delta_pp": 10.574983187626096,
+      "primary_metric": "accuracy",
+      "task": "MassiveIntentClassification",
+      "task_type": "Classification"
+    },
+    {
+      "candidate_100": 76.11297915265635,
+      "current_100": 73.37256220578345,
+      "delta_pp": 2.740416946872898,
+      "primary_metric": "accuracy",
+      "task": "MassiveScenarioClassification",
+      "task_type": "Classification"
+    },
+    {
+      "candidate_100": 39.83023091223438,
+      "current_100": 37.76813963073551,
+      "delta_pp": 2.0620912814988657,
+      "primary_metric": "v_measure",
+      "task": "MedrxivClusteringP2P.v2",
+      "task_type": "Clustering"
+    },
+    {
+      "candidate_100": 37.875084453707764,
+      "current_100": 36.13349306174702,
+      "delta_pp": 1.7415913919607462,
+      "primary_metric": "v_measure",
+      "task": "MedrxivClusteringS2S.v2",
+      "task_type": "Clustering"
+    },
+    {
+      "candidate_100": 32.366,
+      "current_100": 30.623,
+      "delta_pp": 1.7429999999999986,
+      "primary_metric": "max_over_subqueries_map_at_1000",
+      "task": "MindSmallReranking",
+      "task_type": "Reranking"
+    },
+    {
+      "candidate_100": 21.889,
+      "current_100": 16.827,
+      "delta_pp": 5.061999999999998,
+      "primary_metric": "ndcg_at_10",
+      "task": "SCIDOCS",
+      "task_type": "Retrieval"
+    },
+    {
+      "candidate_100": 80.53174565838819,
+      "current_100": 71.69618925026361,
+      "delta_pp": 8.835556408124575,
+      "primary_metric": "cosine_spearman",
+      "task": "SICK-R",
+      "task_type": "STS"
+    },
+    {
+      "candidate_100": 75.5658801516535,
+      "current_100": 68.16915974310523,
+      "delta_pp": 7.396720408548276,
+      "primary_metric": "cosine_spearman",
+      "task": "STS12",
+      "task_type": "STS"
+    },
+    {
+      "candidate_100": 86.26346974038235,
+      "current_100": 74.40342742804673,
+      "delta_pp": 11.860042312335622,
+      "primary_metric": "cosine_spearman",
+      "task": "STS13",
+      "task_type": "STS"
+    },
+    {
+      "candidate_100": 82.29882793900394,
+      "current_100": 68.02653891259911,
+      "delta_pp": 14.272289026404835,
+      "primary_metric": "cosine_spearman",
+      "task": "STS14",
+      "task_type": "STS"
+    },
+    {
+      "candidate_100": 88.73648284326485,
+      "current_100": 77.86144933320074,
+      "delta_pp": 10.875033510064114,
+      "primary_metric": "cosine_spearman",
+      "task": "STS15",
+      "task_type": "STS"
+    },
+    {
+      "candidate_100": 87.0781967302773,
+      "current_100": 77.92400226361768,
+      "delta_pp": 9.154194466659618,
+      "primary_metric": "cosine_spearman",
+      "task": "STSBenchmark",
+      "task_type": "STS"
+    },
+    {
+      "candidate_100": 95.79964673462752,
+      "current_100": 89.79096125036985,
+      "delta_pp": 6.0086854842576685,
+      "primary_metric": "max_ap",
+      "task": "SprintDuplicateQuestions",
+      "task_type": "PairClassification"
+    },
+    {
+      "candidate_100": 58.424418738430106,
+      "current_100": 57.34464990059502,
+      "delta_pp": 1.0797688378350827,
+      "primary_metric": "v_measure",
+      "task": "StackExchangeClustering.v2",
+      "task_type": "Clustering"
+    },
+    {
+      "candidate_100": 41.053017253245386,
+      "current_100": 39.37668627014545,
+      "delta_pp": 1.6763309830999376,
+      "primary_metric": "v_measure",
+      "task": "StackExchangeClusteringP2P.v2",
+      "task_type": "Clustering"
+    },
+    {
+      "candidate_100": 69.121,
+      "current_100": 41.292,
+      "delta_pp": 27.828999999999994,
+      "primary_metric": "ndcg_at_10",
+      "task": "TRECCOVID",
+      "task_type": "Retrieval"
+    },
+    {
+      "candidate_100": 48.363,
+      "current_100": 31.477,
+      "delta_pp": 16.886,
+      "primary_metric": "ndcg_at_10",
+      "task": "Touche2020Retrieval.v3",
+      "task_type": "Retrieval"
+    },
+    {
+      "candidate_100": 71.904296875,
+      "current_100": 64.4091796875,
+      "delta_pp": 7.4951171875,
+      "primary_metric": "accuracy",
+      "task": "ToxicConversationsClassification",
+      "task_type": "Classification"
+    },
+    {
+      "candidate_100": 62.79286926994907,
+      "current_100": 50.77815506508207,
+      "delta_pp": 12.014714204867005,
+      "primary_metric": "accuracy",
+      "task": "TweetSentimentExtractionClassification",
+      "task_type": "Classification"
+    },
+    {
+      "candidate_100": 50.87260708286914,
+      "current_100": 44.88471745749435,
+      "delta_pp": 5.987889625374791,
+      "primary_metric": "v_measure",
+      "task": "TwentyNewsgroupsClustering.v2",
+      "task_type": "Clustering"
+    },
+    {
+      "candidate_100": 72.95151691526065,
+      "current_100": 56.34475441213181,
+      "delta_pp": 16.60676250312884,
+      "primary_metric": "max_ap",
+      "task": "TwitterSemEval2015",
+      "task_type": "PairClassification"
+    },
+    {
+      "candidate_100": 85.30340320492272,
+      "current_100": 78.70209561308617,
+      "delta_pp": 6.6013075918365445,
+      "primary_metric": "max_ap",
+      "task": "TwitterURLCorpus",
+      "task_type": "PairClassification"
+    },
+    {
+      "candidate_100": 31.834434345907127,
+      "current_100": 29.249640152019634,
+      "delta_pp": 2.584794193887493,
+      "primary_metric": "cosine_spearman",
+      "task": "SummEvalSummarization.v2",
+      "task_type": "Summarization"
+    },
+    {
+      "candidate_100": 71.8507462686567,
+      "current_100": 56.82089552238806,
+      "delta_pp": 15.02985074626865,
+      "primary_metric": "accuracy",
+      "task": "AmazonCounterfactualClassification",
+      "task_type": "Classification"
+    },
+    {
+      "candidate_100": 89.02101343562433,
+      "current_100": 84.16589279728525,
+      "delta_pp": 4.8551206383390735,
+      "primary_metric": "cosine_spearman",
+      "task": "STS17",
+      "task_type": "STS"
+    },
+    {
+      "candidate_100": 68.75333638044488,
+      "current_100": 67.57005849656171,
+      "delta_pp": 1.1832778838831643,
+      "primary_metric": "cosine_spearman",
+      "task": "STS22.v2",
+      "task_type": "STS"
+    }
+  ],
+  "audio_task_delta_pp": {
+    "BeijingOpera": 0.0,
+    "BirdCLEF": 0.0,
+    "CREMADPairClassification": 0.12104996267769952,
+    "CREMA_D": 0.0,
+    "CREMA_DClustering": 0.0,
+    "CommonLanguageAgeDetection": 0.0,
+    "GTZANAudioReranking": 0.0,
+    "GTZANGenre": 0.0,
+    "IEMOCAPGender": 0.0,
+    "JamAltArtistA2ARetrieval": 0.0,
+    "MInDS14": 0.0,
+    "MridinghamTonic": -0.14336917562723928,
+    "NMSQAPairClassification": -0.1520590751812989,
+    "SIBFLEURS": 0.0,
+    "VehicleSoundClustering": 0.0,
+    "VoxCelebSA": 0.0,
+    "VoxPopuliAccentPairClassification": -0.02844431688273641,
+    "VoxPopuliGenderClustering": 0.0,
+    "VoxPopuliLanguageID": 0.0
+  },
+  "paired_interval_scope": "Paired intervals cover the original common metrics, family and two-primary-per-family utility. They are not confidence intervals for the new all-14 common mean or the four-component product aggregate."
+}

benchmarks/pareto-gallery.md

@@ -1,50 +1,19 @@
 # Size–quality figures
 
-[Model card](../README.md) · [Detailed evaluation](EVALUATION.md) · [Methodology and sources](pareto-methodology.md) · [All points (CSV)](pareto-points.csv)
+All eligible known-size observations enter each frontier; reported and measured peers remain separate. See [methodology](pareto-methodology.md).
 
-Maintained task-level comparisons from the complete 41-task English and 19-task audio evaluations. Every figure includes both Vela models and all peers with known positive total parameter counts. The September 17, 2026 registry snapshot is retained alongside a separately marked September 18 F2LLM v2 80M rerun. Evaluation protocols can differ.
+## ArXivHierarchicalClusteringP2P
 
-| Task | Metric | Nano | Mini | Vela on observed frontier |
-| --- | --- | ---: | ---: | --- |
-| ArXiv hierarchical clustering | V-measure | 65.25 | 65.52 | — |
-| NMSQA audio pair classification | Maximum average precision | **63.44** | 86.55 | Nano |
-| SIBFLEURS audio classification | Accuracy | 18.30 | **39.43** | Mini |
-| Vehicle sound clustering | V-measure | **13.80** | 8.72 | Nano |
+![ArXivHierarchicalClusteringP2P](../assets/pareto-arxiv.png)
 
-Bold scores identify a Vela model on the observed task frontier. They do not indicate an overall benchmark rank or a statistically significant lead. Vela sizes include all text, image and audio components.
+## NMSQAPairClassification
 
-## ArXiv hierarchical clustering
+![NMSQAPairClassification](../assets/pareto-nmsqa.png)
 
-MTEB English / MMTEB · V-measure · 210 reported peer models + one measured observation
+## SIBFLEURS
 
-[F2LLM v2 80M](peer-evaluations/f2llm-v2-80m-arxiv.json) scores **69.74** in the official MTEB 2.21.0 task protocol. Its separate reported point remains in the figure. Neither Vela model is on the updated ArXiv frontier.
+![SIBFLEURS](../assets/pareto-sibfleurs.png)
 
-![ArXiv hierarchical clustering: total parameters versus V-measure, with the observed Pareto frontier](../assets/pareto-arxiv.png)
+## VehicleSoundClustering
 
-[Vector figure (SVG)](../assets/pareto-arxiv.svg)
-
-## NMSQA audio pair classification
-
-MAEB audio · Maximum average precision · 66 reported peer models
-
-![NMSQA audio pair classification: total parameters versus Maximum average precision, with the observed Pareto frontier](../assets/pareto-nmsqa.png)
-
-[Vector figure (SVG)](../assets/pareto-nmsqa.svg)
-
-## SIBFLEURS audio classification
-
-MAEB audio · Accuracy · 67 reported peer models
-
-![SIBFLEURS audio classification: total parameters versus Accuracy, with the observed Pareto frontier](../assets/pareto-sibfleurs.png)
-
-[Vector figure (SVG)](../assets/pareto-sibfleurs.svg)
-
-## Vehicle sound clustering
-
-MAEB audio · V-measure · 67 reported peer models
-
-![Vehicle sound clustering: total parameters versus V-measure, with the observed Pareto frontier](../assets/pareto-vehiclesoundclustering.png)
-
-[Vector figure (SVG)](../assets/pareto-vehiclesoundclustering.svg)
-
-Vehicle sound clustering has a small observed margin: Nano scores 13.80 versus 13.37 for the strongest reported peer, AST AudioSet (87M). This snapshot comparison does not establish statistical significance.
+![VehicleSoundClustering](../assets/pareto-vehiclesoundclustering.png)

benchmarks/pareto-methodology.md

@@ -1,43 +1,11 @@
-# Size–quality comparison methodology
+# Pareto methodology
 
-[Model card](../README.md) · [Figure gallery](pareto-gallery.md) · [Complete evaluation](EVALUATION.md) · [All plotted points (CSV)](pareto-points.csv) · [Scores, sources and exclusions (JSON)](pareto-data.json)
+All finite task scores with a known positive total parameter count are retained from the September 17, 2026 MTEB API snapshots, including closed-weight models. Their exact source URLs, snapshot hashes and exclusions are in [pareto-data.json](pareto-data.json). The horizontal axis counts the whole stored model, including all three Vela modalities; reported peer counts can be rounded or outdated. No unknown size is imputed. Model-card and registry specifications can differ (for example Jina Text Nano); the registry coordinate is retained rather than silently mixing specifications from another date.
 
-These figures show the observed size–quality trade-off on four individual tasks. These comparisons were initially selected by screening all 41 English and 19 audio-only tasks for both released Vela models. Existing figures are retained when new observations move a model off the frontier. They are not an average across those panels or a complete multilingual, image or multimodal benchmark result.
+The frontier uses every displayed observation: a point is dominated if another is no larger and no lower scoring, with at least one strict improvement. Pairwise dominance and a separately implemented sorted sweep agree. All four plots include both Vela models. Labels are reading aids, not eligibility filters.
 
-| Task | Benchmark family | Main metric | External peers | Vela model on the observed frontier |
-| --- | --- | --- | ---: | --- |
-| ArXivHierarchicalClusteringP2P | MTEB English v2 / MMTEB | V-measure | 210 models / 211 observations | — |
-| NMSQAPairClassification | MAEB audio | Maximum average precision | 66 | Nano |
-| SIBFLEURS | MAEB audio | Accuracy | 67 | Mini |
-| VehicleSoundClustering | MAEB audio | V-measure | 67 | Nano |
+Reported and independently measured observations remain separate, even for the same model. F2's reported size has an explicit checkpoint-based correction in its report; its actual rerun uses the official task instruction, BF16 and an 8192-token limit. AST's old API point has 87M rounded parameters and no revision; it is not silently normalized. The new AST point counts all 86,594,063 parameters of the pinned checkpoint (86,187,264 execute as the encoder) and uses the official AST processor/raw pooler, FP32, batch eight, all 1,705 Vehicle rows and ten default clustering repetitions with seed 20260918. No training or task tuning is involved. The original NumPy/tensor interface failure and the narrow compatible wrapper are retained in its evidence.
 
-## Points and frontier
+Nano's English coordinate retains the actual evaluated artifact and applies through exact text identity. Audio coordinates are current-artifact evaluations. All raw identities, parameters, metrics and rich results remain in the linked reports. Within-task API protocol differences and unknown training overlap limit causal comparison. No official benchmark-wide or size-SOTA claim follows from these four task-level plots.
 
-Each point is a model with a finite score for the exact named task and a known, strictly positive total parameter count. There is no upper size cutoff. Both open and closed weights are included; the two closed-weight ArXiv models use diamond markers. Missing scores and unknown or zero reported sizes cannot be placed on these logarithmic axes and are excluded explicitly in the JSON.
-
-Duplicate registry observations across the English/multilingual and audio-only/full-audio source panels agree in score and size and are represented once per model. Independent reruns remain separate observations with their own identity and protocol, even when the model name matches a registry point. A model need not have completed the entire source panel to contribute its reported score on the named task. There are two additional Vela points on every plot.
-
-A point is Pareto-optimal if no other displayed model has no more parameters and no lower score, with at least one strict improvement. Every eligible point participates in this calculation. Dashed segments join the nondominated observations as a visual guide; they do not predict scores for unmeasured model sizes. Bold colored labels identify Vela. Named references include widely used Qwen, E5, Jina, BGE, Whisper, Wav2Vec2 and CLAP models where scores are available, including models below the frontier. Names are abbreviated for readability. Axis ranges retain every eligible point; the ArXiv and NMSQA vertical axes are cropped to show score differences more clearly. Full model IDs are in the downloads.
-
-## Parameters and scores
-
-The horizontal axis is logarithmic and uses **total parameters**, not active parameters or embedding dimensions. Vela Nano has 133,956,480 parameters and Mini has 1,044,152,640, counting the complete text, image and audio model. Registry peer sizes generally use the snapshot’s rounded `totalParamsB` field. Both F2LLM v2 80M observations use **80,084,288** parameters, verified by each original checkpoint’s model metadata and by counting the rerun checkpoint. The original API size, **80M**, remains in a separate field; no reported score is replaced. This prevents rounding precision from creating an artificial frontier point. These reported values are not an independent parameter audit and may differ from a model card updated at another time; for example, the snapshot reports 212M for Jina embeddings v5 text nano. Point coordinates preserve the snapshot rather than combining specifications from different dates.
-
-The vertical axis uses each task's primary metric multiplied by 100, without transforming or rescaling the scores relative to competitors. SIBFLEURS uses the official labeled classification probe; NMSQA is an audio pair-classification task. These audio results belong to MAEB, not the multilingual text MMTEB benchmark.
-
-Vela points come from the complete [English task report](mteb-eng-v2.json) and [audio task report](maeb-audio-only.json). The reports retain the actual evaluated revisions and processing, including component-equivalent current artifacts. The current Nano image update preserves the exact evaluated text and audio computation; its reports retain the original evaluated identity and explicitly record current applicability. Vela uses MTEB 2.21.0, FP32, batch size 8, seed 20260918 and its native token limits; full protocols are in the [evaluation report](EVALUATION.md).
-
-## Sources and comparison limits
-
-The [MTEB leaderboard](https://leaderboard.mteb.org/) API snapshot was fetched on **September 17, 2026, around 20:28 UTC**. Exact source URLs, fetch timestamps and SHA-256 hashes are included in the JSON. Sources are the English v2 and multilingual v2 panels for ArXiv, and the audio-only and full MAEB beta panels for NMSQA, SIBFLEURS and VehicleSoundClustering.
-
-The registry provides reported scores, not matched reruns of every model. It does not establish identical scored checkpoint or dataset revisions, prompts, precision, seeds, preprocessing or training exposure across peers. Some models use training examples from the evaluated task families; the JSON preserves registered training-task information where available. A position on these observed frontiers supports a task-specific size–quality comparison, not a statistical significance claim, an official rank, or overall SOTA.
-
-
-## F2LLM v2 80M single-task rerun
-
-The [complete peer report](peer-evaluations/f2llm-v2-80m-arxiv.json) retains the exact checkpoint, dataset and software versions, primary and ancillary metrics, and every repeat. The rerun scores **69.7422 V-measure** versus Nano’s **65.2540** and Mini’s **65.5229**. Both reported and rerun observations enter the frontier; neither Vela model is nondominated on ArXiv.
-
-MTEB 2.21.0’s official task first selects 2,048 rows, then its clustering default embeds `int(0.04 × 2048) = 81` items. Each of two label levels uses 10 bootstrap clustering repeats with 16,384 draws with replacement. The reported standard deviation, **3.3618 points**, spans those 20 level/repeat scores and is not a confidence interval. This is the official task default, not an added evaluation cap, and does not encode the full arXiv corpus.
-
-The dataset revision, official evaluator, seed 20260918 and batch size 8 match Vela’s run. F2 uses its official instruction plus EOS, BF16, 320-dimensional outputs and an 8192-token effective limit; Vela Nano uses no task instruction, FP32, 384 dimensions and 512 tokens. The F2 registry declares related ArxivClusteringP2P and ArxivClusteringS2S training families, so this comparison does not establish unseen-task generalization. The original API-reported F2 score remains **63.4236**, at checkpoint `35baae50` using MTEB 2.6.7. The rerun uses checkpoint `6faa7738` and MTEB 2.21.0. The original task file and model metadata are linked in the peer report. Both points use the verified exact parameter count on the horizontal axis; the API’s rounded 80M size is retained separately. This is not a same-checkpoint, same-version comparison.
+[All plotted points (CSV)](pareto-points.csv) · [F2 rerun](peer-evaluations/f2llm-v2-80m-arxiv.json) · [AST rerun](peer-evaluations/ast-vehicle.json) · [Complete evaluation](EVALUATION.md)

benchmarks/peer-evaluations/ast-vehicle.json

@@ -0,0 +1,163 @@
+{
+  "repo": "MIT/ast-finetuned-audioset-10-10-0.4593",
+  "revision": "f826b80d28226b62986cc218e5cec390b1096902",
+  "task": "VehicleSoundClustering",
+  "named_primary": "v_measure",
+  "score": 0.14342872113086985,
+  "score_100": 14.342872113086985,
+  "license": "bsd-3-clause",
+  "parameters_stored_checkpoint": 86594063,
+  "parameters_executed_encoder": 86187264,
+  "all_metrics_and_repeats": {
+    "ami": 0.14330370471140763,
+    "ami_scores": {
+      "Level 0": [
+        0.16483933024740335,
+        0.15862468751797598,
+        0.16459283744076228,
+        0.13045715805289232,
+        0.15083143572383018,
+        0.15421416384956105,
+        0.1604645365220733,
+        0.06431653718837808,
+        0.13333167946336774,
+        0.15136468110783216
+      ]
+    },
+    "ami_std": 0.028618820516157432,
+    "hf_subset": "default",
+    "languages": [
+      "zxx-Zxxx"
+    ],
+    "main_score": 0.14342872113086985,
+    "mteb_version": "2.21.0",
+    "v_measure": 0.14342872113086985,
+    "v_measure_std": 0.028616883697764793,
+    "v_measures": {
+      "Level 0": [
+        0.16496510766521402,
+        0.15874800888615836,
+        0.16471608494012052,
+        0.13058082120519865,
+        0.15095836842024427,
+        0.15433903721698403,
+        0.1605862479922362,
+        0.06444764847146177,
+        0.13345642401427982,
+        0.15148946249680087
+      ]
+    }
+  },
+  "api_reported_score_100": 13.366700000000002,
+  "schema_version": 1,
+  "framework": "MTEB",
+  "framework_version": "2.21.0",
+  "parameter_definition": "All checkpoint parameters (86,594,063); executed encoder has 86,187,264. The classifier is stored but not used.",
+  "dataset": {
+    "path": "mteb/VehicleSoundClustering",
+    "revision": "5f905f9544d0c78b607639662c88c2ddd1216384",
+    "split": "test",
+    "subset": "default",
+    "rows": 1705
+  },
+  "protocol": {
+    "seed": 20260918,
+    "batch_size": 8,
+    "dtype": "float32",
+    "embedding_dimensions": 768,
+    "normalization": "No additional L2; official AST pooler output.",
+    "repetitions": 10,
+    "bootstrap_rows_per_repetition": 16384,
+    "bootstrap_with_replacement": true,
+    "clustering": "Official MiniBatchKMeans, 3 clusters, batch size 512, k-means++, n_init=1, random_state=20260918.",
+    "interface_compatibility": "The MTEB 2.21 wrapper assumes a tensor while AudioCollator supplies NumPy. Only torch.as_tensor was added before the unchanged processor; synthetic CPU parity with the pinned official encoder and processor is exact."
+  },
+  "historical_registry": {
+    "parameters_billion": 0.087,
+    "revision": null,
+    "score": 0.133667,
+    "scope": "Original API observation is retained separately. No revision is available to justify replacing its rounded parameter count with the pinned checkpoint count."
+  },
+  "independent_verification": {
+    "saved_assignments_metrics_exact": true,
+    "bootstrap_repeats": [
+      {
+        "ami": 0.16483933024740335,
+        "bootstrap_indices_sha256": "b4f5ce8ce825948af4363e89b56599bf95ac481f1e3b4bf51d3e2bf06039fb1c",
+        "repeat": 0,
+        "rows": 16384,
+        "v_measure": 0.16496510766521402
+      },
+      {
+        "ami": 0.15862468751797598,
+        "bootstrap_indices_sha256": "d535c32eaec1689ef9e69b0a54366840c372124bc63ec8978839888021c29f3a",
+        "repeat": 1,
+        "rows": 16384,
+        "v_measure": 0.15874800888615836
+      },
+      {
+        "ami": 0.16459283744076228,
+        "bootstrap_indices_sha256": "a15011f26d8fdb160c09f46368dd3514c2b35b97cce54d94bf93d84d20b000eb",
+        "repeat": 2,
+        "rows": 16384,
+        "v_measure": 0.16471608494012052
+      },
+      {
+        "ami": 0.13045715805289232,
+        "bootstrap_indices_sha256": "0695435ef48fdd3fd975bb0f1ee573ee78850e2234f72a58296cedd9db2dbf30",
+        "repeat": 3,
+        "rows": 16384,
+        "v_measure": 0.13058082120519865
+      },
+      {
+        "ami": 0.15083143572383018,
+        "bootstrap_indices_sha256": "58a12318ee9e76da312f6fe8842801c37cee8365b18a01e29d949ebe2bca7fe9",
+        "repeat": 4,
+        "rows": 16384,
+        "v_measure": 0.15095836842024427
+      },
+      {
+        "ami": 0.15421416384956105,
+        "bootstrap_indices_sha256": "389f4cf752b49e35d2a9f1ad47957afdb903afa57a3fd40e31c672e4682ce68c",
+        "repeat": 5,
+        "rows": 16384,
+        "v_measure": 0.15433903721698403
+      },
+      {
+        "ami": 0.1604645365220733,
+        "bootstrap_indices_sha256": "5f876025683dd22d544757bc3d3b99e377861db880610500de9bb397fcb8228c",
+        "repeat": 6,
+        "rows": 16384,
+        "v_measure": 0.1605862479922362
+      },
+      {
+        "ami": 0.06431653718837808,
+        "bootstrap_indices_sha256": "a63d85c311cb13dbc84194e4cc7ed7a34be8c93367fdae8cba0315bf17141ecb",
+        "repeat": 7,
+        "rows": 16384,
+        "v_measure": 0.06444764847146177
+      },
+      {
+        "ami": 0.13333167946336774,
+        "bootstrap_indices_sha256": "2760c005b7b6fc80beefac6d536b4cd90eb4f47572dccc3fe3fd8354427033c9",
+        "repeat": 8,
+        "rows": 16384,
+        "v_measure": 0.13345642401427982
+      },
+      {
+        "ami": 0.15136468110783216,
+        "bootstrap_indices_sha256": "7fa60e3596dca79c20ceebf1868e404125ae8a3dfba999e00e9a6fe5e7bfeeef",
+        "repeat": 9,
+        "rows": 16384,
+        "v_measure": 0.15148946249680087
+      }
+    ],
+    "reconciliation_sha256": "9025a50919dced96c0582696a3dd5e28619eb3a5981ab249a528d4307d6390e9",
+    "raw_result_sha256": "000762c7f259c681e44f77e763ea3ec4824ed1e578d358da1a8531a22f0b11ba"
+  },
+  "sources": [
+    "https://huggingface.co/MIT/ast-finetuned-audioset-10-10-0.4593/tree/f826b80d28226b62986cc218e5cec390b1096902",
+    "https://github.com/YuanGongND/ast"
+  ],
+  "scope": "A specialist peer rerun on one official task; not an overall multimodal comparison or training result."
+}

components/text/1_Pooling/config.json

@@ -1,7 +1,7 @@
 {
   "word_embedding_dimension": 384,
-  "pooling_mode_cls_token": false,
-  "pooling_mode_mean_tokens": true,
+  "pooling_mode_cls_token": true,
+  "pooling_mode_mean_tokens": false,
   "pooling_mode_max_tokens": false,
   "pooling_mode_mean_sqrt_len_tokens": false
-}
+}

components/text/config.json

@@ -1,22 +1,30 @@
 {
+  "_name_or_path": "avsolatorio/GIST-small-Embedding-v0",
   "architectures": [
     "BertModel"
   ],
   "attention_probs_dropout_prob": 0.1,
-  "gradient_checkpointing": false,
+  "classifier_dropout": null,
   "hidden_act": "gelu",
   "hidden_dropout_prob": 0.1,
   "hidden_size": 384,
+  "id2label": {
+    "0": "LABEL_0"
+  },
   "initializer_range": 0.02,
   "intermediate_size": 1536,
+  "label2id": {
+    "LABEL_0": 0
+  },
   "layer_norm_eps": 1e-12,
   "max_position_embeddings": 512,
   "model_type": "bert",
   "num_attention_heads": 12,
-  "num_hidden_layers": 6,
+  "num_hidden_layers": 12,
   "pad_token_id": 0,
   "position_embedding_type": "absolute",
-  "transformers_version": "4.8.2",
+  "torch_dtype": "float32",
+  "transformers_version": "4.37.2",
   "type_vocab_size": 2,
   "use_cache": true,
   "vocab_size": 30522

components/text/config_sentence_transformers.json

@@ -0,0 +1,7 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.2.2",
+    "transformers": "4.28.1",
+    "pytorch": "1.13.0+cu117"
+  }
+}

components/text/modules.json

@@ -17,4 +17,4 @@
     "path": "2_Normalize",
     "type": "sentence_transformers.models.Normalize"
   }
-]
+]

components/text/sentence_bert_config.json

@@ -1,4 +1,4 @@
 {
-  "max_seq_length": 256,
-  "do_lower_case": false
-}
+  "max_seq_length": 512,
+  "do_lower_case": true
+}

components/text/special_tokens_map.json

@@ -1,7 +1,37 @@
 {
-  "unk_token": "[UNK]",
-  "sep_token": "[SEP]",
-  "pad_token": "[PAD]",
-  "cls_token": "[CLS]",
-  "mask_token": "[MASK]"
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
 }

components/text/tokenizer_config.json

@@ -1,14 +1,57 @@
 {
-  "do_lower_case": true,
-  "unk_token": "[UNK]",
-  "sep_token": "[SEP]",
-  "pad_token": "[PAD]",
+  "added_tokens_decoder": {
+    "0": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "100": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "101": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "102": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "103": {
+      "content": "[MASK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "clean_up_tokenization_spaces": true,
   "cls_token": "[CLS]",
-  "mask_token": "[MASK]",
-  "tokenize_chinese_chars": true,
-  "strip_accents": null,
   "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "mask_token": "[MASK]",
+  "model_max_length": 512,
   "never_split": null,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
   "tokenizer_class": "BertTokenizer",
-  "model_max_length": 512
+  "unk_token": "[UNK]"
 }

config.json

@@ -1,7 +1,7 @@
 {
-  "format_version": 3,
+  "format_version": 4,
   "variant": "nano",
-  "parameter_count": 133956480,
+  "parameter_count": 144554112,
   "embedding_dim": 384,
   "max_text_length": 512,
   "architectures": [
@@ -12,6 +12,7 @@
   "audio_sampling_rate": 16000,
   "audio_max_seconds": 30,
   "audio_projection": "linear",
-  "text_projection": "residual_rank64",
-  "audio_pooling": "mean"
+  "text_projection": "identity",
+  "audio_pooling": "mean",
+  "text_pooling": "cls"
 }

licenses/GIST-MODEL-CARD.md

@@ -0,0 +1,2702 @@
+---
+language:
+- en
+library_name: sentence-transformers
+license: mit
+pipeline_tag: sentence-similarity
+tags:
+  - feature-extraction
+  - mteb
+  - sentence-similarity
+  - sentence-transformers
+
+model-index:
+- name: GIST-small-Embedding-v0
+  results:
+  - task:
+      type: Classification
+    dataset:
+      type: mteb/amazon_counterfactual
+      name: MTEB AmazonCounterfactualClassification (en)
+      config: en
+      split: test
+      revision: e8379541af4e31359cca9fbcf4b00f2671dba205
+    metrics:
+    - type: accuracy
+      value: 75.26865671641791
+    - type: ap
+      value: 38.25623793370476
+    - type: f1
+      value: 69.26434651320257
+  - task:
+      type: Classification
+    dataset:
+      type: mteb/amazon_polarity
+      name: MTEB AmazonPolarityClassification
+      config: default
+      split: test
+      revision: e2d317d38cd51312af73b3d32a06d1a08b442046
+    metrics:
+    - type: accuracy
+      value: 93.232225
+    - type: ap
+      value: 89.97936072879344
+    - type: f1
+      value: 93.22122653806187
+  - task:
+      type: Classification
+    dataset:
+      type: mteb/amazon_reviews_multi
+      name: MTEB AmazonReviewsClassification (en)
+      config: en
+      split: test
+      revision: 1399c76144fd37290681b995c656ef9b2e06e26d
+    metrics:
+    - type: accuracy
+      value: 49.715999999999994
+    - type: f1
+      value: 49.169789920136076
+  - task:
+      type: Retrieval
+    dataset:
+      type: arguana
+      name: MTEB ArguAna
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 34.922
+    - type: map_at_10
+      value: 50.524
+    - type: map_at_100
+      value: 51.247
+    - type: map_at_1000
+      value: 51.249
+    - type: map_at_3
+      value: 45.887
+    - type: map_at_5
+      value: 48.592999999999996
+    - type: mrr_at_1
+      value: 34.922
+    - type: mrr_at_10
+      value: 50.382000000000005
+    - type: mrr_at_100
+      value: 51.104000000000006
+    - type: mrr_at_1000
+      value: 51.105999999999995
+    - type: mrr_at_3
+      value: 45.733000000000004
+    - type: mrr_at_5
+      value: 48.428
+    - type: ndcg_at_1
+      value: 34.922
+    - type: ndcg_at_10
+      value: 59.12
+    - type: ndcg_at_100
+      value: 62.083999999999996
+    - type: ndcg_at_1000
+      value: 62.137
+    - type: ndcg_at_3
+      value: 49.616
+    - type: ndcg_at_5
+      value: 54.501
+    - type: precision_at_1
+      value: 34.922
+    - type: precision_at_10
+      value: 8.649
+    - type: precision_at_100
+      value: 0.991
+    - type: precision_at_1000
+      value: 0.1
+    - type: precision_at_3
+      value: 20.152
+    - type: precision_at_5
+      value: 14.466999999999999
+    - type: recall_at_1
+      value: 34.922
+    - type: recall_at_10
+      value: 86.48599999999999
+    - type: recall_at_100
+      value: 99.14699999999999
+    - type: recall_at_1000
+      value: 99.57300000000001
+    - type: recall_at_3
+      value: 60.455000000000005
+    - type: recall_at_5
+      value: 72.333
+  - task:
+      type: Clustering
+    dataset:
+      type: mteb/arxiv-clustering-p2p
+      name: MTEB ArxivClusteringP2P
+      config: default
+      split: test
+      revision: a122ad7f3f0291bf49cc6f4d32aa80929df69d5d
+    metrics:
+    - type: v_measure
+      value: 47.623282347623714
+  - task:
+      type: Clustering
+    dataset:
+      type: mteb/arxiv-clustering-s2s
+      name: MTEB ArxivClusteringS2S
+      config: default
+      split: test
+      revision: f910caf1a6075f7329cdf8c1a6135696f37dbd53
+    metrics:
+    - type: v_measure
+      value: 39.86487843524932
+  - task:
+      type: Reranking
+    dataset:
+      type: mteb/askubuntudupquestions-reranking
+      name: MTEB AskUbuntuDupQuestions
+      config: default
+      split: test
+      revision: 2000358ca161889fa9c082cb41daa8dcfb161a54
+    metrics:
+    - type: map
+      value: 62.3290291318171
+    - type: mrr
+      value: 75.2379853141626
+  - task:
+      type: STS
+    dataset:
+      type: mteb/biosses-sts
+      name: MTEB BIOSSES
+      config: default
+      split: test
+      revision: d3fb88f8f02e40887cd149695127462bbcf29b4a
+    metrics:
+    - type: cos_sim_pearson
+      value: 88.52002953574285
+    - type: cos_sim_spearman
+      value: 86.98752423842483
+    - type: euclidean_pearson
+      value: 86.89442688314197
+    - type: euclidean_spearman
+      value: 86.88631711307471
+    - type: manhattan_pearson
+      value: 87.03723618507175
+    - type: manhattan_spearman
+      value: 86.76041062975224
+  - task:
+      type: Classification
+    dataset:
+      type: mteb/banking77
+      name: MTEB Banking77Classification
+      config: default
+      split: test
+      revision: 0fd18e25b25c072e09e0d92ab615fda904d66300
+    metrics:
+    - type: accuracy
+      value: 86.64935064935065
+    - type: f1
+      value: 86.61903824934998
+  - task:
+      type: Clustering
+    dataset:
+      type: mteb/biorxiv-clustering-p2p
+      name: MTEB BiorxivClusteringP2P
+      config: default
+      split: test
+      revision: 65b79d1d13f80053f67aca9498d9402c2d9f1f40
+    metrics:
+    - type: v_measure
+      value: 39.21904455377494
+  - task:
+      type: Clustering
+    dataset:
+      type: mteb/biorxiv-clustering-s2s
+      name: MTEB BiorxivClusteringS2S
+      config: default
+      split: test
+      revision: 258694dd0231531bc1fd9de6ceb52a0853c6d908
+    metrics:
+    - type: v_measure
+      value: 35.43342755570654
+  - task:
+      type: Retrieval
+    dataset:
+      type: BeIR/cqadupstack
+      name: MTEB CQADupstackAndroidRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 31.843
+    - type: map_at_10
+      value: 43.379
+    - type: map_at_100
+      value: 44.946999999999996
+    - type: map_at_1000
+      value: 45.078
+    - type: map_at_3
+      value: 39.598
+    - type: map_at_5
+      value: 41.746
+    - type: mrr_at_1
+      value: 39.199
+    - type: mrr_at_10
+      value: 49.672
+    - type: mrr_at_100
+      value: 50.321000000000005
+    - type: mrr_at_1000
+      value: 50.365
+    - type: mrr_at_3
+      value: 46.805
+    - type: mrr_at_5
+      value: 48.579
+    - type: ndcg_at_1
+      value: 39.199
+    - type: ndcg_at_10
+      value: 50.163999999999994
+    - type: ndcg_at_100
+      value: 55.418
+    - type: ndcg_at_1000
+      value: 57.353
+    - type: ndcg_at_3
+      value: 44.716
+    - type: ndcg_at_5
+      value: 47.268
+    - type: precision_at_1
+      value: 39.199
+    - type: precision_at_10
+      value: 9.757
+    - type: precision_at_100
+      value: 1.552
+    - type: precision_at_1000
+      value: 0.20500000000000002
+    - type: precision_at_3
+      value: 21.602
+    - type: precision_at_5
+      value: 15.479000000000001
+    - type: recall_at_1
+      value: 31.843
+    - type: recall_at_10
+      value: 62.743
+    - type: recall_at_100
+      value: 84.78099999999999
+    - type: recall_at_1000
+      value: 96.86099999999999
+    - type: recall_at_3
+      value: 46.927
+    - type: recall_at_5
+      value: 54.355
+  - task:
+      type: Retrieval
+    dataset:
+      type: BeIR/cqadupstack
+      name: MTEB CQADupstackEnglishRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 29.321
+    - type: map_at_10
+      value: 39.062999999999995
+    - type: map_at_100
+      value: 40.403
+    - type: map_at_1000
+      value: 40.534
+    - type: map_at_3
+      value: 36.367
+    - type: map_at_5
+      value: 37.756
+    - type: mrr_at_1
+      value: 35.987
+    - type: mrr_at_10
+      value: 44.708999999999996
+    - type: mrr_at_100
+      value: 45.394
+    - type: mrr_at_1000
+      value: 45.436
+    - type: mrr_at_3
+      value: 42.463
+    - type: mrr_at_5
+      value: 43.663000000000004
+    - type: ndcg_at_1
+      value: 35.987
+    - type: ndcg_at_10
+      value: 44.585
+    - type: ndcg_at_100
+      value: 49.297999999999995
+    - type: ndcg_at_1000
+      value: 51.315
+    - type: ndcg_at_3
+      value: 40.569
+    - type: ndcg_at_5
+      value: 42.197
+    - type: precision_at_1
+      value: 35.987
+    - type: precision_at_10
+      value: 8.369
+    - type: precision_at_100
+      value: 1.366
+    - type: precision_at_1000
+      value: 0.184
+    - type: precision_at_3
+      value: 19.427
+    - type: precision_at_5
+      value: 13.58
+    - type: recall_at_1
+      value: 29.321
+    - type: recall_at_10
+      value: 54.333
+    - type: recall_at_100
+      value: 74.178
+    - type: recall_at_1000
+      value: 86.732
+    - type: recall_at_3
+      value: 42.46
+    - type: recall_at_5
+      value: 47.089999999999996
+  - task:
+      type: Retrieval
+    dataset:
+      type: BeIR/cqadupstack
+      name: MTEB CQADupstackGamingRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 38.811
+    - type: map_at_10
+      value: 51.114000000000004
+    - type: map_at_100
+      value: 52.22
+    - type: map_at_1000
+      value: 52.275000000000006
+    - type: map_at_3
+      value: 47.644999999999996
+    - type: map_at_5
+      value: 49.675000000000004
+    - type: mrr_at_1
+      value: 44.389
+    - type: mrr_at_10
+      value: 54.459
+    - type: mrr_at_100
+      value: 55.208999999999996
+    - type: mrr_at_1000
+      value: 55.239000000000004
+    - type: mrr_at_3
+      value: 51.954
+    - type: mrr_at_5
+      value: 53.571999999999996
+    - type: ndcg_at_1
+      value: 44.389
+    - type: ndcg_at_10
+      value: 56.979
+    - type: ndcg_at_100
+      value: 61.266
+    - type: ndcg_at_1000
+      value: 62.315
+    - type: ndcg_at_3
+      value: 51.342
+    - type: ndcg_at_5
+      value: 54.33
+    - type: precision_at_1
+      value: 44.389
+    - type: precision_at_10
+      value: 9.26
+    - type: precision_at_100
+      value: 1.226
+    - type: precision_at_1000
+      value: 0.136
+    - type: precision_at_3
+      value: 22.926
+    - type: precision_at_5
+      value: 15.987000000000002
+    - type: recall_at_1
+      value: 38.811
+    - type: recall_at_10
+      value: 70.841
+    - type: recall_at_100
+      value: 89.218
+    - type: recall_at_1000
+      value: 96.482
+    - type: recall_at_3
+      value: 56.123999999999995
+    - type: recall_at_5
+      value: 63.322
+  - task:
+      type: Retrieval
+    dataset:
+      type: BeIR/cqadupstack
+      name: MTEB CQADupstackGisRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 25.378
+    - type: map_at_10
+      value: 34.311
+    - type: map_at_100
+      value: 35.399
+    - type: map_at_1000
+      value: 35.482
+    - type: map_at_3
+      value: 31.917
+    - type: map_at_5
+      value: 33.275
+    - type: mrr_at_1
+      value: 27.683999999999997
+    - type: mrr_at_10
+      value: 36.575
+    - type: mrr_at_100
+      value: 37.492
+    - type: mrr_at_1000
+      value: 37.556
+    - type: mrr_at_3
+      value: 34.35
+    - type: mrr_at_5
+      value: 35.525
+    - type: ndcg_at_1
+      value: 27.683999999999997
+    - type: ndcg_at_10
+      value: 39.247
+    - type: ndcg_at_100
+      value: 44.424
+    - type: ndcg_at_1000
+      value: 46.478
+    - type: ndcg_at_3
+      value: 34.684
+    - type: ndcg_at_5
+      value: 36.886
+    - type: precision_at_1
+      value: 27.683999999999997
+    - type: precision_at_10
+      value: 5.989
+    - type: precision_at_100
+      value: 0.899
+    - type: precision_at_1000
+      value: 0.11199999999999999
+    - type: precision_at_3
+      value: 14.84
+    - type: precision_at_5
+      value: 10.215
+    - type: recall_at_1
+      value: 25.378
+    - type: recall_at_10
+      value: 52.195
+    - type: recall_at_100
+      value: 75.764
+    - type: recall_at_1000
+      value: 91.012
+    - type: recall_at_3
+      value: 39.885999999999996
+    - type: recall_at_5
+      value: 45.279
+  - task:
+      type: Retrieval
+    dataset:
+      type: BeIR/cqadupstack
+      name: MTEB CQADupstackMathematicaRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 17.326
+    - type: map_at_10
+      value: 25.247000000000003
+    - type: map_at_100
+      value: 26.473000000000003
+    - type: map_at_1000
+      value: 26.579000000000004
+    - type: map_at_3
+      value: 22.466
+    - type: map_at_5
+      value: 24.113
+    - type: mrr_at_1
+      value: 21.393
+    - type: mrr_at_10
+      value: 30.187
+    - type: mrr_at_100
+      value: 31.089
+    - type: mrr_at_1000
+      value: 31.15
+    - type: mrr_at_3
+      value: 27.279999999999998
+    - type: mrr_at_5
+      value: 29.127
+    - type: ndcg_at_1
+      value: 21.393
+    - type: ndcg_at_10
+      value: 30.668
+    - type: ndcg_at_100
+      value: 36.543
+    - type: ndcg_at_1000
+      value: 39.181
+    - type: ndcg_at_3
+      value: 25.552000000000003
+    - type: ndcg_at_5
+      value: 28.176000000000002
+    - type: precision_at_1
+      value: 21.393
+    - type: precision_at_10
+      value: 5.784000000000001
+    - type: precision_at_100
+      value: 1.001
+    - type: precision_at_1000
+      value: 0.136
+    - type: precision_at_3
+      value: 12.231
+    - type: precision_at_5
+      value: 9.179
+    - type: recall_at_1
+      value: 17.326
+    - type: recall_at_10
+      value: 42.415000000000006
+    - type: recall_at_100
+      value: 68.605
+    - type: recall_at_1000
+      value: 87.694
+    - type: recall_at_3
+      value: 28.343
+    - type: recall_at_5
+      value: 35.086
+  - task:
+      type: Retrieval
+    dataset:
+      type: BeIR/cqadupstack
+      name: MTEB CQADupstackPhysicsRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 29.069
+    - type: map_at_10
+      value: 40.027
+    - type: map_at_100
+      value: 41.308
+    - type: map_at_1000
+      value: 41.412
+    - type: map_at_3
+      value: 36.864000000000004
+    - type: map_at_5
+      value: 38.641999999999996
+    - type: mrr_at_1
+      value: 35.707
+    - type: mrr_at_10
+      value: 45.527
+    - type: mrr_at_100
+      value: 46.348
+    - type: mrr_at_1000
+      value: 46.392
+    - type: mrr_at_3
+      value: 43.086
+    - type: mrr_at_5
+      value: 44.645
+    - type: ndcg_at_1
+      value: 35.707
+    - type: ndcg_at_10
+      value: 46.117000000000004
+    - type: ndcg_at_100
+      value: 51.468
+    - type: ndcg_at_1000
+      value: 53.412000000000006
+    - type: ndcg_at_3
+      value: 41.224
+    - type: ndcg_at_5
+      value: 43.637
+    - type: precision_at_1
+      value: 35.707
+    - type: precision_at_10
+      value: 8.459999999999999
+    - type: precision_at_100
+      value: 1.2970000000000002
+    - type: precision_at_1000
+      value: 0.165
+    - type: precision_at_3
+      value: 19.731
+    - type: precision_at_5
+      value: 14.013
+    - type: recall_at_1
+      value: 29.069
+    - type: recall_at_10
+      value: 58.343999999999994
+    - type: recall_at_100
+      value: 81.296
+    - type: recall_at_1000
+      value: 93.974
+    - type: recall_at_3
+      value: 44.7
+    - type: recall_at_5
+      value: 50.88700000000001
+  - task:
+      type: Retrieval
+    dataset:
+      type: BeIR/cqadupstack
+      name: MTEB CQADupstackProgrammersRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 23.905
+    - type: map_at_10
+      value: 33.983000000000004
+    - type: map_at_100
+      value: 35.372
+    - type: map_at_1000
+      value: 35.487
+    - type: map_at_3
+      value: 30.902
+    - type: map_at_5
+      value: 32.505
+    - type: mrr_at_1
+      value: 29.794999999999998
+    - type: mrr_at_10
+      value: 39.28
+    - type: mrr_at_100
+      value: 40.215
+    - type: mrr_at_1000
+      value: 40.276
+    - type: mrr_at_3
+      value: 36.701
+    - type: mrr_at_5
+      value: 38.105
+    - type: ndcg_at_1
+      value: 29.794999999999998
+    - type: ndcg_at_10
+      value: 40.041
+    - type: ndcg_at_100
+      value: 45.884
+    - type: ndcg_at_1000
+      value: 48.271
+    - type: ndcg_at_3
+      value: 34.931
+    - type: ndcg_at_5
+      value: 37.044
+    - type: precision_at_1
+      value: 29.794999999999998
+    - type: precision_at_10
+      value: 7.546
+    - type: precision_at_100
+      value: 1.216
+    - type: precision_at_1000
+      value: 0.158
+    - type: precision_at_3
+      value: 16.933
+    - type: precision_at_5
+      value: 12.1
+    - type: recall_at_1
+      value: 23.905
+    - type: recall_at_10
+      value: 52.945
+    - type: recall_at_100
+      value: 77.551
+    - type: recall_at_1000
+      value: 93.793
+    - type: recall_at_3
+      value: 38.364
+    - type: recall_at_5
+      value: 44.044
+  - task:
+      type: Retrieval
+    dataset:
+      type: BeIR/cqadupstack
+      name: MTEB CQADupstackRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 25.24441666666667
+    - type: map_at_10
+      value: 34.4595
+    - type: map_at_100
+      value: 35.699999999999996
+    - type: map_at_1000
+      value: 35.8155
+    - type: map_at_3
+      value: 31.608333333333338
+    - type: map_at_5
+      value: 33.189416666666666
+    - type: mrr_at_1
+      value: 29.825250000000004
+    - type: mrr_at_10
+      value: 38.60875
+    - type: mrr_at_100
+      value: 39.46575
+    - type: mrr_at_1000
+      value: 39.52458333333333
+    - type: mrr_at_3
+      value: 36.145166666666675
+    - type: mrr_at_5
+      value: 37.57625
+    - type: ndcg_at_1
+      value: 29.825250000000004
+    - type: ndcg_at_10
+      value: 39.88741666666667
+    - type: ndcg_at_100
+      value: 45.17966666666667
+    - type: ndcg_at_1000
+      value: 47.440583333333336
+    - type: ndcg_at_3
+      value: 35.04591666666666
+    - type: ndcg_at_5
+      value: 37.32025
+    - type: precision_at_1
+      value: 29.825250000000004
+    - type: precision_at_10
+      value: 7.07225
+    - type: precision_at_100
+      value: 1.1462499999999998
+    - type: precision_at_1000
+      value: 0.15325
+    - type: precision_at_3
+      value: 16.18375
+    - type: precision_at_5
+      value: 11.526833333333334
+    - type: recall_at_1
+      value: 25.24441666666667
+    - type: recall_at_10
+      value: 51.744916666666676
+    - type: recall_at_100
+      value: 75.04574999999998
+    - type: recall_at_1000
+      value: 90.65558333333334
+    - type: recall_at_3
+      value: 38.28349999999999
+    - type: recall_at_5
+      value: 44.16591666666667
+  - task:
+      type: Retrieval
+    dataset:
+      type: BeIR/cqadupstack
+      name: MTEB CQADupstackStatsRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 24.237000000000002
+    - type: map_at_10
+      value: 30.667
+    - type: map_at_100
+      value: 31.592
+    - type: map_at_1000
+      value: 31.688
+    - type: map_at_3
+      value: 28.810999999999996
+    - type: map_at_5
+      value: 29.788999999999998
+    - type: mrr_at_1
+      value: 26.840000000000003
+    - type: mrr_at_10
+      value: 33.305
+    - type: mrr_at_100
+      value: 34.089000000000006
+    - type: mrr_at_1000
+      value: 34.159
+    - type: mrr_at_3
+      value: 31.518
+    - type: mrr_at_5
+      value: 32.469
+    - type: ndcg_at_1
+      value: 26.840000000000003
+    - type: ndcg_at_10
+      value: 34.541
+    - type: ndcg_at_100
+      value: 39.206
+    - type: ndcg_at_1000
+      value: 41.592
+    - type: ndcg_at_3
+      value: 31.005
+    - type: ndcg_at_5
+      value: 32.554
+    - type: precision_at_1
+      value: 26.840000000000003
+    - type: precision_at_10
+      value: 5.3069999999999995
+    - type: precision_at_100
+      value: 0.8340000000000001
+    - type: precision_at_1000
+      value: 0.11199999999999999
+    - type: precision_at_3
+      value: 13.292000000000002
+    - type: precision_at_5
+      value: 9.049
+    - type: recall_at_1
+      value: 24.237000000000002
+    - type: recall_at_10
+      value: 43.862
+    - type: recall_at_100
+      value: 65.352
+    - type: recall_at_1000
+      value: 82.704
+    - type: recall_at_3
+      value: 34.009
+    - type: recall_at_5
+      value: 37.878
+  - task:
+      type: Retrieval
+    dataset:
+      type: BeIR/cqadupstack
+      name: MTEB CQADupstackTexRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 16.482
+    - type: map_at_10
+      value: 23.249
+    - type: map_at_100
+      value: 24.388
+    - type: map_at_1000
+      value: 24.519
+    - type: map_at_3
+      value: 20.971
+    - type: map_at_5
+      value: 22.192
+    - type: mrr_at_1
+      value: 19.993
+    - type: mrr_at_10
+      value: 26.985
+    - type: mrr_at_100
+      value: 27.975
+    - type: mrr_at_1000
+      value: 28.052
+    - type: mrr_at_3
+      value: 24.954
+    - type: mrr_at_5
+      value: 26.070999999999998
+    - type: ndcg_at_1
+      value: 19.993
+    - type: ndcg_at_10
+      value: 27.656
+    - type: ndcg_at_100
+      value: 33.256
+    - type: ndcg_at_1000
+      value: 36.275
+    - type: ndcg_at_3
+      value: 23.644000000000002
+    - type: ndcg_at_5
+      value: 25.466
+    - type: precision_at_1
+      value: 19.993
+    - type: precision_at_10
+      value: 5.093
+    - type: precision_at_100
+      value: 0.932
+    - type: precision_at_1000
+      value: 0.13699999999999998
+    - type: precision_at_3
+      value: 11.149000000000001
+    - type: precision_at_5
+      value: 8.149000000000001
+    - type: recall_at_1
+      value: 16.482
+    - type: recall_at_10
+      value: 37.141999999999996
+    - type: recall_at_100
+      value: 62.696
+    - type: recall_at_1000
+      value: 84.333
+    - type: recall_at_3
+      value: 26.031
+    - type: recall_at_5
+      value: 30.660999999999998
+  - task:
+      type: Retrieval
+    dataset:
+      type: BeIR/cqadupstack
+      name: MTEB CQADupstackUnixRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 24.887999999999998
+    - type: map_at_10
+      value: 34.101
+    - type: map_at_100
+      value: 35.27
+    - type: map_at_1000
+      value: 35.370000000000005
+    - type: map_at_3
+      value: 31.283
+    - type: map_at_5
+      value: 32.72
+    - type: mrr_at_1
+      value: 29.011
+    - type: mrr_at_10
+      value: 38.004
+    - type: mrr_at_100
+      value: 38.879000000000005
+    - type: mrr_at_1000
+      value: 38.938
+    - type: mrr_at_3
+      value: 35.571999999999996
+    - type: mrr_at_5
+      value: 36.789
+    - type: ndcg_at_1
+      value: 29.011
+    - type: ndcg_at_10
+      value: 39.586
+    - type: ndcg_at_100
+      value: 44.939
+    - type: ndcg_at_1000
+      value: 47.236
+    - type: ndcg_at_3
+      value: 34.4
+    - type: ndcg_at_5
+      value: 36.519
+    - type: precision_at_1
+      value: 29.011
+    - type: precision_at_10
+      value: 6.763
+    - type: precision_at_100
+      value: 1.059
+    - type: precision_at_1000
+      value: 0.13699999999999998
+    - type: precision_at_3
+      value: 15.609
+    - type: precision_at_5
+      value: 10.896
+    - type: recall_at_1
+      value: 24.887999999999998
+    - type: recall_at_10
+      value: 52.42
+    - type: recall_at_100
+      value: 75.803
+    - type: recall_at_1000
+      value: 91.725
+    - type: recall_at_3
+      value: 38.080999999999996
+    - type: recall_at_5
+      value: 43.47
+  - task:
+      type: Retrieval
+    dataset:
+      type: BeIR/cqadupstack
+      name: MTEB CQADupstackWebmastersRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 23.953
+    - type: map_at_10
+      value: 32.649
+    - type: map_at_100
+      value: 34.181
+    - type: map_at_1000
+      value: 34.398
+    - type: map_at_3
+      value: 29.567
+    - type: map_at_5
+      value: 31.263
+    - type: mrr_at_1
+      value: 29.051
+    - type: mrr_at_10
+      value: 37.419999999999995
+    - type: mrr_at_100
+      value: 38.396
+    - type: mrr_at_1000
+      value: 38.458
+    - type: mrr_at_3
+      value: 34.782999999999994
+    - type: mrr_at_5
+      value: 36.254999999999995
+    - type: ndcg_at_1
+      value: 29.051
+    - type: ndcg_at_10
+      value: 38.595
+    - type: ndcg_at_100
+      value: 44.6
+    - type: ndcg_at_1000
+      value: 47.158
+    - type: ndcg_at_3
+      value: 33.56
+    - type: ndcg_at_5
+      value: 35.870000000000005
+    - type: precision_at_1
+      value: 29.051
+    - type: precision_at_10
+      value: 7.53
+    - type: precision_at_100
+      value: 1.538
+    - type: precision_at_1000
+      value: 0.24
+    - type: precision_at_3
+      value: 15.744
+    - type: precision_at_5
+      value: 11.542
+    - type: recall_at_1
+      value: 23.953
+    - type: recall_at_10
+      value: 50.08200000000001
+    - type: recall_at_100
+      value: 77.364
+    - type: recall_at_1000
+      value: 93.57799999999999
+    - type: recall_at_3
+      value: 35.432
+    - type: recall_at_5
+      value: 41.875
+  - task:
+      type: Retrieval
+    dataset:
+      type: BeIR/cqadupstack
+      name: MTEB CQADupstackWordpressRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 17.72
+    - type: map_at_10
+      value: 25.724000000000004
+    - type: map_at_100
+      value: 26.846999999999998
+    - type: map_at_1000
+      value: 26.964
+    - type: map_at_3
+      value: 22.909
+    - type: map_at_5
+      value: 24.596999999999998
+    - type: mrr_at_1
+      value: 18.854000000000003
+    - type: mrr_at_10
+      value: 27.182000000000002
+    - type: mrr_at_100
+      value: 28.182000000000002
+    - type: mrr_at_1000
+      value: 28.274
+    - type: mrr_at_3
+      value: 24.276
+    - type: mrr_at_5
+      value: 26.115
+    - type: ndcg_at_1
+      value: 18.854000000000003
+    - type: ndcg_at_10
+      value: 30.470000000000002
+    - type: ndcg_at_100
+      value: 35.854
+    - type: ndcg_at_1000
+      value: 38.701
+    - type: ndcg_at_3
+      value: 24.924
+    - type: ndcg_at_5
+      value: 27.895999999999997
+    - type: precision_at_1
+      value: 18.854000000000003
+    - type: precision_at_10
+      value: 5.009
+    - type: precision_at_100
+      value: 0.835
+    - type: precision_at_1000
+      value: 0.117
+    - type: precision_at_3
+      value: 10.721
+    - type: precision_at_5
+      value: 8.133
+    - type: recall_at_1
+      value: 17.72
+    - type: recall_at_10
+      value: 43.617
+    - type: recall_at_100
+      value: 67.941
+    - type: recall_at_1000
+      value: 88.979
+    - type: recall_at_3
+      value: 29.044999999999998
+    - type: recall_at_5
+      value: 36.044
+  - task:
+      type: Retrieval
+    dataset:
+      type: climate-fever
+      name: MTEB ClimateFEVER
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 13.427
+    - type: map_at_10
+      value: 22.935
+    - type: map_at_100
+      value: 24.808
+    - type: map_at_1000
+      value: 24.994
+    - type: map_at_3
+      value: 19.533
+    - type: map_at_5
+      value: 21.261
+    - type: mrr_at_1
+      value: 30.945
+    - type: mrr_at_10
+      value: 43.242000000000004
+    - type: mrr_at_100
+      value: 44.013999999999996
+    - type: mrr_at_1000
+      value: 44.048
+    - type: mrr_at_3
+      value: 40.109
+    - type: mrr_at_5
+      value: 42.059999999999995
+    - type: ndcg_at_1
+      value: 30.945
+    - type: ndcg_at_10
+      value: 31.828
+    - type: ndcg_at_100
+      value: 38.801
+    - type: ndcg_at_1000
+      value: 42.126999999999995
+    - type: ndcg_at_3
+      value: 26.922
+    - type: ndcg_at_5
+      value: 28.483999999999998
+    - type: precision_at_1
+      value: 30.945
+    - type: precision_at_10
+      value: 9.844
+    - type: precision_at_100
+      value: 1.7309999999999999
+    - type: precision_at_1000
+      value: 0.23500000000000001
+    - type: precision_at_3
+      value: 20.477999999999998
+    - type: precision_at_5
+      value: 15.27
+    - type: recall_at_1
+      value: 13.427
+    - type: recall_at_10
+      value: 37.141000000000005
+    - type: recall_at_100
+      value: 61.007
+    - type: recall_at_1000
+      value: 79.742
+    - type: recall_at_3
+      value: 24.431
+    - type: recall_at_5
+      value: 29.725
+  - task:
+      type: Retrieval
+    dataset:
+      type: dbpedia-entity
+      name: MTEB DBPedia
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 9.122
+    - type: map_at_10
+      value: 18.799
+    - type: map_at_100
+      value: 25.724999999999998
+    - type: map_at_1000
+      value: 27.205000000000002
+    - type: map_at_3
+      value: 14.194999999999999
+    - type: map_at_5
+      value: 16.225
+    - type: mrr_at_1
+      value: 68.0
+    - type: mrr_at_10
+      value: 76.035
+    - type: mrr_at_100
+      value: 76.292
+    - type: mrr_at_1000
+      value: 76.297
+    - type: mrr_at_3
+      value: 74.458
+    - type: mrr_at_5
+      value: 75.558
+    - type: ndcg_at_1
+      value: 56.00000000000001
+    - type: ndcg_at_10
+      value: 39.761
+    - type: ndcg_at_100
+      value: 43.736999999999995
+    - type: ndcg_at_1000
+      value: 51.146
+    - type: ndcg_at_3
+      value: 45.921
+    - type: ndcg_at_5
+      value: 42.756
+    - type: precision_at_1
+      value: 68.0
+    - type: precision_at_10
+      value: 30.275000000000002
+    - type: precision_at_100
+      value: 9.343
+    - type: precision_at_1000
+      value: 1.8270000000000002
+    - type: precision_at_3
+      value: 49.167
+    - type: precision_at_5
+      value: 40.699999999999996
+    - type: recall_at_1
+      value: 9.122
+    - type: recall_at_10
+      value: 23.669999999999998
+    - type: recall_at_100
+      value: 48.719
+    - type: recall_at_1000
+      value: 72.033
+    - type: recall_at_3
+      value: 15.498999999999999
+    - type: recall_at_5
+      value: 18.657
+  - task:
+      type: Classification
+    dataset:
+      type: mteb/emotion
+      name: MTEB EmotionClassification
+      config: default
+      split: test
+      revision: 4f58c6b202a23cf9a4da393831edf4f9183cad37
+    metrics:
+    - type: accuracy
+      value: 55.885000000000005
+    - type: f1
+      value: 50.70726446938571
+  - task:
+      type: Retrieval
+    dataset:
+      type: fever
+      name: MTEB FEVER
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 75.709
+    - type: map_at_10
+      value: 83.345
+    - type: map_at_100
+      value: 83.557
+    - type: map_at_1000
+      value: 83.572
+    - type: map_at_3
+      value: 82.425
+    - type: map_at_5
+      value: 83.013
+    - type: mrr_at_1
+      value: 81.593
+    - type: mrr_at_10
+      value: 88.331
+    - type: mrr_at_100
+      value: 88.408
+    - type: mrr_at_1000
+      value: 88.41
+    - type: mrr_at_3
+      value: 87.714
+    - type: mrr_at_5
+      value: 88.122
+    - type: ndcg_at_1
+      value: 81.593
+    - type: ndcg_at_10
+      value: 86.925
+    - type: ndcg_at_100
+      value: 87.67
+    - type: ndcg_at_1000
+      value: 87.924
+    - type: ndcg_at_3
+      value: 85.5
+    - type: ndcg_at_5
+      value: 86.283
+    - type: precision_at_1
+      value: 81.593
+    - type: precision_at_10
+      value: 10.264
+    - type: precision_at_100
+      value: 1.084
+    - type: precision_at_1000
+      value: 0.11199999999999999
+    - type: precision_at_3
+      value: 32.388
+    - type: precision_at_5
+      value: 19.991
+    - type: recall_at_1
+      value: 75.709
+    - type: recall_at_10
+      value: 93.107
+    - type: recall_at_100
+      value: 96.024
+    - type: recall_at_1000
+      value: 97.603
+    - type: recall_at_3
+      value: 89.08500000000001
+    - type: recall_at_5
+      value: 91.15299999999999
+  - task:
+      type: Retrieval
+    dataset:
+      type: fiqa
+      name: MTEB FiQA2018
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 19.121
+    - type: map_at_10
+      value: 31.78
+    - type: map_at_100
+      value: 33.497
+    - type: map_at_1000
+      value: 33.696
+    - type: map_at_3
+      value: 27.893
+    - type: map_at_5
+      value: 30.087000000000003
+    - type: mrr_at_1
+      value: 38.272
+    - type: mrr_at_10
+      value: 47.176
+    - type: mrr_at_100
+      value: 48.002
+    - type: mrr_at_1000
+      value: 48.044
+    - type: mrr_at_3
+      value: 45.086999999999996
+    - type: mrr_at_5
+      value: 46.337
+    - type: ndcg_at_1
+      value: 38.272
+    - type: ndcg_at_10
+      value: 39.145
+    - type: ndcg_at_100
+      value: 45.696999999999996
+    - type: ndcg_at_1000
+      value: 49.0
+    - type: ndcg_at_3
+      value: 36.148
+    - type: ndcg_at_5
+      value: 37.023
+    - type: precision_at_1
+      value: 38.272
+    - type: precision_at_10
+      value: 11.065
+    - type: precision_at_100
+      value: 1.7840000000000003
+    - type: precision_at_1000
+      value: 0.23600000000000002
+    - type: precision_at_3
+      value: 24.587999999999997
+    - type: precision_at_5
+      value: 18.056
+    - type: recall_at_1
+      value: 19.121
+    - type: recall_at_10
+      value: 44.857
+    - type: recall_at_100
+      value: 69.774
+    - type: recall_at_1000
+      value: 89.645
+    - type: recall_at_3
+      value: 32.588
+    - type: recall_at_5
+      value: 37.939
+  - task:
+      type: Retrieval
+    dataset:
+      type: hotpotqa
+      name: MTEB HotpotQA
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 36.428
+    - type: map_at_10
+      value: 56.891999999999996
+    - type: map_at_100
+      value: 57.82899999999999
+    - type: map_at_1000
+      value: 57.896
+    - type: map_at_3
+      value: 53.762
+    - type: map_at_5
+      value: 55.718
+    - type: mrr_at_1
+      value: 72.856
+    - type: mrr_at_10
+      value: 79.245
+    - type: mrr_at_100
+      value: 79.515
+    - type: mrr_at_1000
+      value: 79.525
+    - type: mrr_at_3
+      value: 78.143
+    - type: mrr_at_5
+      value: 78.822
+    - type: ndcg_at_1
+      value: 72.856
+    - type: ndcg_at_10
+      value: 65.204
+    - type: ndcg_at_100
+      value: 68.552
+    - type: ndcg_at_1000
+      value: 69.902
+    - type: ndcg_at_3
+      value: 60.632
+    - type: ndcg_at_5
+      value: 63.161
+    - type: precision_at_1
+      value: 72.856
+    - type: precision_at_10
+      value: 13.65
+    - type: precision_at_100
+      value: 1.6260000000000001
+    - type: precision_at_1000
+      value: 0.181
+    - type: precision_at_3
+      value: 38.753
+    - type: precision_at_5
+      value: 25.251
+    - type: recall_at_1
+      value: 36.428
+    - type: recall_at_10
+      value: 68.25099999999999
+    - type: recall_at_100
+      value: 81.317
+    - type: recall_at_1000
+      value: 90.27
+    - type: recall_at_3
+      value: 58.13
+    - type: recall_at_5
+      value: 63.126000000000005
+  - task:
+      type: Classification
+    dataset:
+      type: mteb/imdb
+      name: MTEB ImdbClassification
+      config: default
+      split: test
+      revision: 3d86128a09e091d6018b6d26cad27f2739fc2db7
+    metrics:
+    - type: accuracy
+      value: 89.4868
+    - type: ap
+      value: 84.88319192880247
+    - type: f1
+      value: 89.46144458052846
+  - task:
+      type: Retrieval
+    dataset:
+      type: msmarco
+      name: MTEB MSMARCO
+      config: default
+      split: dev
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 21.282999999999998
+    - type: map_at_10
+      value: 33.045
+    - type: map_at_100
+      value: 34.238
+    - type: map_at_1000
+      value: 34.29
+    - type: map_at_3
+      value: 29.305999999999997
+    - type: map_at_5
+      value: 31.391000000000002
+    - type: mrr_at_1
+      value: 21.92
+    - type: mrr_at_10
+      value: 33.649
+    - type: mrr_at_100
+      value: 34.791
+    - type: mrr_at_1000
+      value: 34.837
+    - type: mrr_at_3
+      value: 30.0
+    - type: mrr_at_5
+      value: 32.039
+    - type: ndcg_at_1
+      value: 21.92
+    - type: ndcg_at_10
+      value: 39.729
+    - type: ndcg_at_100
+      value: 45.484
+    - type: ndcg_at_1000
+      value: 46.817
+    - type: ndcg_at_3
+      value: 32.084
+    - type: ndcg_at_5
+      value: 35.789
+    - type: precision_at_1
+      value: 21.92
+    - type: precision_at_10
+      value: 6.297
+    - type: precision_at_100
+      value: 0.918
+    - type: precision_at_1000
+      value: 0.10300000000000001
+    - type: precision_at_3
+      value: 13.639000000000001
+    - type: precision_at_5
+      value: 10.054
+    - type: recall_at_1
+      value: 21.282999999999998
+    - type: recall_at_10
+      value: 60.343999999999994
+    - type: recall_at_100
+      value: 86.981
+    - type: recall_at_1000
+      value: 97.205
+    - type: recall_at_3
+      value: 39.452999999999996
+    - type: recall_at_5
+      value: 48.333
+  - task:
+      type: Classification
+    dataset:
+      type: mteb/mtop_domain
+      name: MTEB MTOPDomainClassification (en)
+      config: en
+      split: test
+      revision: d80d48c1eb48d3562165c59d59d0034df9fff0bf
+    metrics:
+    - type: accuracy
+      value: 95.47879616963064
+    - type: f1
+      value: 95.21800589958251
+  - task:
+      type: Classification
+    dataset:
+      type: mteb/mtop_intent
+      name: MTEB MTOPIntentClassification (en)
+      config: en
+      split: test
+      revision: ae001d0e6b1228650b7bd1c2c65fb50ad11a8aba
+    metrics:
+    - type: accuracy
+      value: 79.09256725946192
+    - type: f1
+      value: 60.554043889452515
+  - task:
+      type: Classification
+    dataset:
+      type: mteb/amazon_massive_intent
+      name: MTEB MassiveIntentClassification (en)
+      config: en
+      split: test
+      revision: 31efe3c427b0bae9c22cbb560b8f15491cc6bed7
+    metrics:
+    - type: accuracy
+      value: 75.53463349024882
+    - type: f1
+      value: 73.14418495756476
+  - task:
+      type: Classification
+    dataset:
+      type: mteb/amazon_massive_scenario
+      name: MTEB MassiveScenarioClassification (en)
+      config: en
+      split: test
+      revision: 7d571f92784cd94a019292a1f45445077d0ef634
+    metrics:
+    - type: accuracy
+      value: 79.22663080026899
+    - type: f1
+      value: 79.331456217501
+  - task:
+      type: Clustering
+    dataset:
+      type: mteb/medrxiv-clustering-p2p
+      name: MTEB MedrxivClusteringP2P
+      config: default
+      split: test
+      revision: e7a26af6f3ae46b30dde8737f02c07b1505bcc73
+    metrics:
+    - type: v_measure
+      value: 34.50316010430136
+  - task:
+      type: Clustering
+    dataset:
+      type: mteb/medrxiv-clustering-s2s
+      name: MTEB MedrxivClusteringS2S
+      config: default
+      split: test
+      revision: 35191c8c0dca72d8ff3efcd72aa802307d469663
+    metrics:
+    - type: v_measure
+      value: 32.15612040042282
+  - task:
+      type: Reranking
+    dataset:
+      type: mteb/mind_small
+      name: MTEB MindSmallReranking
+      config: default
+      split: test
+      revision: 3bdac13927fdc888b903db93b2ffdbd90b295a69
+    metrics:
+    - type: map
+      value: 32.36227552557184
+    - type: mrr
+      value: 33.57901344209811
+  - task:
+      type: Retrieval
+    dataset:
+      type: nfcorpus
+      name: MTEB NFCorpus
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 5.6610000000000005
+    - type: map_at_10
+      value: 12.992
+    - type: map_at_100
+      value: 16.756999999999998
+    - type: map_at_1000
+      value: 18.25
+    - type: map_at_3
+      value: 9.471
+    - type: map_at_5
+      value: 11.116
+    - type: mrr_at_1
+      value: 43.653
+    - type: mrr_at_10
+      value: 53.388999999999996
+    - type: mrr_at_100
+      value: 53.982
+    - type: mrr_at_1000
+      value: 54.033
+    - type: mrr_at_3
+      value: 51.858000000000004
+    - type: mrr_at_5
+      value: 53.019000000000005
+    - type: ndcg_at_1
+      value: 41.641
+    - type: ndcg_at_10
+      value: 34.691
+    - type: ndcg_at_100
+      value: 32.305
+    - type: ndcg_at_1000
+      value: 41.132999999999996
+    - type: ndcg_at_3
+      value: 40.614
+    - type: ndcg_at_5
+      value: 38.456
+    - type: precision_at_1
+      value: 43.344
+    - type: precision_at_10
+      value: 25.881999999999998
+    - type: precision_at_100
+      value: 8.483
+    - type: precision_at_1000
+      value: 2.131
+    - type: precision_at_3
+      value: 38.803
+    - type: precision_at_5
+      value: 33.87
+    - type: recall_at_1
+      value: 5.6610000000000005
+    - type: recall_at_10
+      value: 16.826
+    - type: recall_at_100
+      value: 32.939
+    - type: recall_at_1000
+      value: 65.161
+    - type: recall_at_3
+      value: 10.756
+    - type: recall_at_5
+      value: 13.331000000000001
+  - task:
+      type: Retrieval
+    dataset:
+      type: nq
+      name: MTEB NQ
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 26.692
+    - type: map_at_10
+      value: 41.065000000000005
+    - type: map_at_100
+      value: 42.235
+    - type: map_at_1000
+      value: 42.27
+    - type: map_at_3
+      value: 36.635
+    - type: map_at_5
+      value: 39.219
+    - type: mrr_at_1
+      value: 30.214000000000002
+    - type: mrr_at_10
+      value: 43.443
+    - type: mrr_at_100
+      value: 44.326
+    - type: mrr_at_1000
+      value: 44.352000000000004
+    - type: mrr_at_3
+      value: 39.623999999999995
+    - type: mrr_at_5
+      value: 41.898
+    - type: ndcg_at_1
+      value: 30.214000000000002
+    - type: ndcg_at_10
+      value: 48.692
+    - type: ndcg_at_100
+      value: 53.671
+    - type: ndcg_at_1000
+      value: 54.522000000000006
+    - type: ndcg_at_3
+      value: 40.245
+    - type: ndcg_at_5
+      value: 44.580999999999996
+    - type: precision_at_1
+      value: 30.214000000000002
+    - type: precision_at_10
+      value: 8.3
+    - type: precision_at_100
+      value: 1.1079999999999999
+    - type: precision_at_1000
+      value: 0.11900000000000001
+    - type: precision_at_3
+      value: 18.521
+    - type: precision_at_5
+      value: 13.627
+    - type: recall_at_1
+      value: 26.692
+    - type: recall_at_10
+      value: 69.699
+    - type: recall_at_100
+      value: 91.425
+    - type: recall_at_1000
+      value: 97.78099999999999
+    - type: recall_at_3
+      value: 47.711
+    - type: recall_at_5
+      value: 57.643
+  - task:
+      type: Retrieval
+    dataset:
+      type: quora
+      name: MTEB QuoraRetrieval
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 70.962
+    - type: map_at_10
+      value: 84.772
+    - type: map_at_100
+      value: 85.402
+    - type: map_at_1000
+      value: 85.418
+    - type: map_at_3
+      value: 81.89
+    - type: map_at_5
+      value: 83.685
+    - type: mrr_at_1
+      value: 81.67
+    - type: mrr_at_10
+      value: 87.681
+    - type: mrr_at_100
+      value: 87.792
+    - type: mrr_at_1000
+      value: 87.79299999999999
+    - type: mrr_at_3
+      value: 86.803
+    - type: mrr_at_5
+      value: 87.392
+    - type: ndcg_at_1
+      value: 81.69
+    - type: ndcg_at_10
+      value: 88.429
+    - type: ndcg_at_100
+      value: 89.66
+    - type: ndcg_at_1000
+      value: 89.762
+    - type: ndcg_at_3
+      value: 85.75
+    - type: ndcg_at_5
+      value: 87.20700000000001
+    - type: precision_at_1
+      value: 81.69
+    - type: precision_at_10
+      value: 13.395000000000001
+    - type: precision_at_100
+      value: 1.528
+    - type: precision_at_1000
+      value: 0.157
+    - type: precision_at_3
+      value: 37.507000000000005
+    - type: precision_at_5
+      value: 24.614
+    - type: recall_at_1
+      value: 70.962
+    - type: recall_at_10
+      value: 95.339
+    - type: recall_at_100
+      value: 99.543
+    - type: recall_at_1000
+      value: 99.984
+    - type: recall_at_3
+      value: 87.54899999999999
+    - type: recall_at_5
+      value: 91.726
+  - task:
+      type: Clustering
+    dataset:
+      type: mteb/reddit-clustering
+      name: MTEB RedditClustering
+      config: default
+      split: test
+      revision: 24640382cdbf8abc73003fb0fa6d111a705499eb
+    metrics:
+    - type: v_measure
+      value: 55.506631779239555
+  - task:
+      type: Clustering
+    dataset:
+      type: mteb/reddit-clustering-p2p
+      name: MTEB RedditClusteringP2P
+      config: default
+      split: test
+      revision: 282350215ef01743dc01b456c7f5241fa8937f16
+    metrics:
+    - type: v_measure
+      value: 60.63731341848479
+  - task:
+      type: Retrieval
+    dataset:
+      type: scidocs
+      name: MTEB SCIDOCS
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 4.852
+    - type: map_at_10
+      value: 13.175
+    - type: map_at_100
+      value: 15.623999999999999
+    - type: map_at_1000
+      value: 16.002
+    - type: map_at_3
+      value: 9.103
+    - type: map_at_5
+      value: 11.068999999999999
+    - type: mrr_at_1
+      value: 23.9
+    - type: mrr_at_10
+      value: 35.847
+    - type: mrr_at_100
+      value: 36.968
+    - type: mrr_at_1000
+      value: 37.018
+    - type: mrr_at_3
+      value: 32.300000000000004
+    - type: mrr_at_5
+      value: 34.14
+    - type: ndcg_at_1
+      value: 23.9
+    - type: ndcg_at_10
+      value: 21.889
+    - type: ndcg_at_100
+      value: 30.903000000000002
+    - type: ndcg_at_1000
+      value: 36.992000000000004
+    - type: ndcg_at_3
+      value: 20.274
+    - type: ndcg_at_5
+      value: 17.773
+    - type: precision_at_1
+      value: 23.9
+    - type: precision_at_10
+      value: 11.61
+    - type: precision_at_100
+      value: 2.4539999999999997
+    - type: precision_at_1000
+      value: 0.391
+    - type: precision_at_3
+      value: 19.133
+    - type: precision_at_5
+      value: 15.740000000000002
+    - type: recall_at_1
+      value: 4.852
+    - type: recall_at_10
+      value: 23.507
+    - type: recall_at_100
+      value: 49.775000000000006
+    - type: recall_at_1000
+      value: 79.308
+    - type: recall_at_3
+      value: 11.637
+    - type: recall_at_5
+      value: 15.947
+  - task:
+      type: STS
+    dataset:
+      type: mteb/sickr-sts
+      name: MTEB SICK-R
+      config: default
+      split: test
+      revision: a6ea5a8cab320b040a23452cc28066d9beae2cee
+    metrics:
+    - type: cos_sim_pearson
+      value: 86.03345827446948
+    - type: cos_sim_spearman
+      value: 80.53174518259549
+    - type: euclidean_pearson
+      value: 83.44538971660883
+    - type: euclidean_spearman
+      value: 80.57344324098692
+    - type: manhattan_pearson
+      value: 83.36528808195459
+    - type: manhattan_spearman
+      value: 80.48931287157902
+  - task:
+      type: STS
+    dataset:
+      type: mteb/sts12-sts
+      name: MTEB STS12
+      config: default
+      split: test
+      revision: a0d554a64d88156834ff5ae9920b964011b16384
+    metrics:
+    - type: cos_sim_pearson
+      value: 85.21363088257881
+    - type: cos_sim_spearman
+      value: 75.56589127055523
+    - type: euclidean_pearson
+      value: 82.32868324521908
+    - type: euclidean_spearman
+      value: 75.31928550664554
+    - type: manhattan_pearson
+      value: 82.31332875713211
+    - type: manhattan_spearman
+      value: 75.35376322099196
+  - task:
+      type: STS
+    dataset:
+      type: mteb/sts13-sts
+      name: MTEB STS13
+      config: default
+      split: test
+      revision: 7e90230a92c190f1bf69ae9002b8cea547a64cca
+    metrics:
+    - type: cos_sim_pearson
+      value: 85.09085593258487
+    - type: cos_sim_spearman
+      value: 86.26355088415221
+    - type: euclidean_pearson
+      value: 85.49646115361156
+    - type: euclidean_spearman
+      value: 86.20652472228703
+    - type: manhattan_pearson
+      value: 85.44084081123815
+    - type: manhattan_spearman
+      value: 86.1162623448951
+  - task:
+      type: STS
+    dataset:
+      type: mteb/sts14-sts
+      name: MTEB STS14
+      config: default
+      split: test
+      revision: 6031580fec1f6af667f0bd2da0a551cf4f0b2375
+    metrics:
+    - type: cos_sim_pearson
+      value: 84.68250248349368
+    - type: cos_sim_spearman
+      value: 82.29883673695083
+    - type: euclidean_pearson
+      value: 84.17633035446019
+    - type: euclidean_spearman
+      value: 82.19990511264791
+    - type: manhattan_pearson
+      value: 84.17408410692279
+    - type: manhattan_spearman
+      value: 82.249873895981
+  - task:
+      type: STS
+    dataset:
+      type: mteb/sts15-sts
+      name: MTEB STS15
+      config: default
+      split: test
+      revision: ae752c7c21bf194d8b67fd573edf7ae58183cbe3
+    metrics:
+    - type: cos_sim_pearson
+      value: 87.31878760045024
+    - type: cos_sim_spearman
+      value: 88.7364409031183
+    - type: euclidean_pearson
+      value: 88.230537618603
+    - type: euclidean_spearman
+      value: 88.76484309646318
+    - type: manhattan_pearson
+      value: 88.17689071136469
+    - type: manhattan_spearman
+      value: 88.72809249037928
+  - task:
+      type: STS
+    dataset:
+      type: mteb/sts16-sts
+      name: MTEB STS16
+      config: default
+      split: test
+      revision: 4d8694f8f0e0100860b497b999b3dbed754a0513
+    metrics:
+    - type: cos_sim_pearson
+      value: 83.41078559110638
+    - type: cos_sim_spearman
+      value: 85.27439135411049
+    - type: euclidean_pearson
+      value: 84.5333571592088
+    - type: euclidean_spearman
+      value: 85.25645460575957
+    - type: manhattan_pearson
+      value: 84.38428921610226
+    - type: manhattan_spearman
+      value: 85.07796040798796
+  - task:
+      type: STS
+    dataset:
+      type: mteb/sts17-crosslingual-sts
+      name: MTEB STS17 (en-en)
+      config: en-en
+      split: test
+      revision: af5e6fb845001ecf41f4c1e033ce921939a2a68d
+    metrics:
+    - type: cos_sim_pearson
+      value: 88.82374132382576
+    - type: cos_sim_spearman
+      value: 89.02101343562433
+    - type: euclidean_pearson
+      value: 89.50729765458932
+    - type: euclidean_spearman
+      value: 89.04184772869253
+    - type: manhattan_pearson
+      value: 89.51737904059856
+    - type: manhattan_spearman
+      value: 89.12925950440676
+  - task:
+      type: STS
+    dataset:
+      type: mteb/sts22-crosslingual-sts
+      name: MTEB STS22 (en)
+      config: en
+      split: test
+      revision: 6d1ba47164174a496b7fa5d3569dae26a6813b80
+    metrics:
+    - type: cos_sim_pearson
+      value: 67.56051823873482
+    - type: cos_sim_spearman
+      value: 68.50988748185463
+    - type: euclidean_pearson
+      value: 69.16524346147456
+    - type: euclidean_spearman
+      value: 68.61859952449579
+    - type: manhattan_pearson
+      value: 69.10618915706995
+    - type: manhattan_spearman
+      value: 68.36401769459522
+  - task:
+      type: STS
+    dataset:
+      type: mteb/stsbenchmark-sts
+      name: MTEB STSBenchmark
+      config: default
+      split: test
+      revision: b0fddb56ed78048fa8b90373c8a3cfc37b684831
+    metrics:
+    - type: cos_sim_pearson
+      value: 85.4159693872625
+    - type: cos_sim_spearman
+      value: 87.07819121764247
+    - type: euclidean_pearson
+      value: 87.03013260863153
+    - type: euclidean_spearman
+      value: 87.06547293631309
+    - type: manhattan_pearson
+      value: 86.8129744446062
+    - type: manhattan_spearman
+      value: 86.88494096335627
+  - task:
+      type: Reranking
+    dataset:
+      type: mteb/scidocs-reranking
+      name: MTEB SciDocsRR
+      config: default
+      split: test
+      revision: d3c5e1fc0b855ab6097bf1cda04dd73947d7caab
+    metrics:
+    - type: map
+      value: 86.47758088996575
+    - type: mrr
+      value: 96.17891458577733
+  - task:
+      type: Retrieval
+    dataset:
+      type: scifact
+      name: MTEB SciFact
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 57.538999999999994
+    - type: map_at_10
+      value: 66.562
+    - type: map_at_100
+      value: 67.254
+    - type: map_at_1000
+      value: 67.284
+    - type: map_at_3
+      value: 63.722
+    - type: map_at_5
+      value: 65.422
+    - type: mrr_at_1
+      value: 60.0
+    - type: mrr_at_10
+      value: 67.354
+    - type: mrr_at_100
+      value: 67.908
+    - type: mrr_at_1000
+      value: 67.93299999999999
+    - type: mrr_at_3
+      value: 65.056
+    - type: mrr_at_5
+      value: 66.43900000000001
+    - type: ndcg_at_1
+      value: 60.0
+    - type: ndcg_at_10
+      value: 70.858
+    - type: ndcg_at_100
+      value: 73.67099999999999
+    - type: ndcg_at_1000
+      value: 74.26700000000001
+    - type: ndcg_at_3
+      value: 65.911
+    - type: ndcg_at_5
+      value: 68.42200000000001
+    - type: precision_at_1
+      value: 60.0
+    - type: precision_at_10
+      value: 9.4
+    - type: precision_at_100
+      value: 1.083
+    - type: precision_at_1000
+      value: 0.11299999999999999
+    - type: precision_at_3
+      value: 25.444
+    - type: precision_at_5
+      value: 17.0
+    - type: recall_at_1
+      value: 57.538999999999994
+    - type: recall_at_10
+      value: 83.233
+    - type: recall_at_100
+      value: 95.667
+    - type: recall_at_1000
+      value: 100.0
+    - type: recall_at_3
+      value: 69.883
+    - type: recall_at_5
+      value: 76.19399999999999
+  - task:
+      type: PairClassification
+    dataset:
+      type: mteb/sprintduplicatequestions-pairclassification
+      name: MTEB SprintDuplicateQuestions
+      config: default
+      split: test
+      revision: d66bd1f72af766a5cc4b0ca5e00c162f89e8cc46
+    metrics:
+    - type: cos_sim_accuracy
+      value: 99.82574257425742
+    - type: cos_sim_ap
+      value: 95.78722833053911
+    - type: cos_sim_f1
+      value: 90.94650205761316
+    - type: cos_sim_precision
+      value: 93.64406779661016
+    - type: cos_sim_recall
+      value: 88.4
+    - type: dot_accuracy
+      value: 99.83366336633664
+    - type: dot_ap
+      value: 95.89733601612964
+    - type: dot_f1
+      value: 91.41981613891727
+    - type: dot_precision
+      value: 93.42379958246346
+    - type: dot_recall
+      value: 89.5
+    - type: euclidean_accuracy
+      value: 99.82574257425742
+    - type: euclidean_ap
+      value: 95.75227035138846
+    - type: euclidean_f1
+      value: 90.96509240246407
+    - type: euclidean_precision
+      value: 93.45991561181435
+    - type: euclidean_recall
+      value: 88.6
+    - type: manhattan_accuracy
+      value: 99.82574257425742
+    - type: manhattan_ap
+      value: 95.76278266220176
+    - type: manhattan_f1
+      value: 91.08409321175279
+    - type: manhattan_precision
+      value: 92.29979466119097
+    - type: manhattan_recall
+      value: 89.9
+    - type: max_accuracy
+      value: 99.83366336633664
+    - type: max_ap
+      value: 95.89733601612964
+    - type: max_f1
+      value: 91.41981613891727
+  - task:
+      type: Clustering
+    dataset:
+      type: mteb/stackexchange-clustering
+      name: MTEB StackExchangeClustering
+      config: default
+      split: test
+      revision: 6cbc1f7b2bc0622f2e39d2c77fa502909748c259
+    metrics:
+    - type: v_measure
+      value: 61.905425988638605
+  - task:
+      type: Clustering
+    dataset:
+      type: mteb/stackexchange-clustering-p2p
+      name: MTEB StackExchangeClusteringP2P
+      config: default
+      split: test
+      revision: 815ca46b2622cec33ccafc3735d572c266efdb44
+    metrics:
+    - type: v_measure
+      value: 36.159589881679736
+  - task:
+      type: Reranking
+    dataset:
+      type: mteb/stackoverflowdupquestions-reranking
+      name: MTEB StackOverflowDupQuestions
+      config: default
+      split: test
+      revision: e185fbe320c72810689fc5848eb6114e1ef5ec69
+    metrics:
+    - type: map
+      value: 53.0605499476397
+    - type: mrr
+      value: 53.91594516594517
+  - task:
+      type: Summarization
+    dataset:
+      type: mteb/summeval
+      name: MTEB SummEval
+      config: default
+      split: test
+      revision: cda12ad7615edc362dbf25a00fdd61d3b1eaf93c
+    metrics:
+    - type: cos_sim_pearson
+      value: 30.202718009067
+    - type: cos_sim_spearman
+      value: 31.136199912366987
+    - type: dot_pearson
+      value: 30.66329011927951
+    - type: dot_spearman
+      value: 30.107664909625107
+  - task:
+      type: Retrieval
+    dataset:
+      type: trec-covid
+      name: MTEB TRECCOVID
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 0.209
+    - type: map_at_10
+      value: 1.712
+    - type: map_at_100
+      value: 9.464
+    - type: map_at_1000
+      value: 23.437
+    - type: map_at_3
+      value: 0.609
+    - type: map_at_5
+      value: 0.9440000000000001
+    - type: mrr_at_1
+      value: 78.0
+    - type: mrr_at_10
+      value: 86.833
+    - type: mrr_at_100
+      value: 86.833
+    - type: mrr_at_1000
+      value: 86.833
+    - type: mrr_at_3
+      value: 85.333
+    - type: mrr_at_5
+      value: 86.833
+    - type: ndcg_at_1
+      value: 74.0
+    - type: ndcg_at_10
+      value: 69.14
+    - type: ndcg_at_100
+      value: 53.047999999999995
+    - type: ndcg_at_1000
+      value: 48.577
+    - type: ndcg_at_3
+      value: 75.592
+    - type: ndcg_at_5
+      value: 72.509
+    - type: precision_at_1
+      value: 78.0
+    - type: precision_at_10
+      value: 73.0
+    - type: precision_at_100
+      value: 54.44
+    - type: precision_at_1000
+      value: 21.326
+    - type: precision_at_3
+      value: 80.667
+    - type: precision_at_5
+      value: 77.2
+    - type: recall_at_1
+      value: 0.209
+    - type: recall_at_10
+      value: 1.932
+    - type: recall_at_100
+      value: 13.211999999999998
+    - type: recall_at_1000
+      value: 45.774
+    - type: recall_at_3
+      value: 0.644
+    - type: recall_at_5
+      value: 1.0290000000000001
+  - task:
+      type: Retrieval
+    dataset:
+      type: webis-touche2020
+      name: MTEB Touche2020
+      config: default
+      split: test
+      revision: None
+    metrics:
+    - type: map_at_1
+      value: 2.609
+    - type: map_at_10
+      value: 8.334999999999999
+    - type: map_at_100
+      value: 14.604000000000001
+    - type: map_at_1000
+      value: 16.177
+    - type: map_at_3
+      value: 4.87
+    - type: map_at_5
+      value: 6.3149999999999995
+    - type: mrr_at_1
+      value: 32.653
+    - type: mrr_at_10
+      value: 45.047
+    - type: mrr_at_100
+      value: 45.808
+    - type: mrr_at_1000
+      value: 45.808
+    - type: mrr_at_3
+      value: 41.497
+    - type: mrr_at_5
+      value: 43.231
+    - type: ndcg_at_1
+      value: 30.612000000000002
+    - type: ndcg_at_10
+      value: 21.193
+    - type: ndcg_at_100
+      value: 34.97
+    - type: ndcg_at_1000
+      value: 46.69
+    - type: ndcg_at_3
+      value: 24.823
+    - type: ndcg_at_5
+      value: 22.872999999999998
+    - type: precision_at_1
+      value: 32.653
+    - type: precision_at_10
+      value: 17.959
+    - type: precision_at_100
+      value: 7.4079999999999995
+    - type: precision_at_1000
+      value: 1.537
+    - type: precision_at_3
+      value: 25.85
+    - type: precision_at_5
+      value: 22.448999999999998
+    - type: recall_at_1
+      value: 2.609
+    - type: recall_at_10
+      value: 13.63
+    - type: recall_at_100
+      value: 47.014
+    - type: recall_at_1000
+      value: 83.176
+    - type: recall_at_3
+      value: 5.925
+    - type: recall_at_5
+      value: 8.574
+  - task:
+      type: Classification
+    dataset:
+      type: mteb/toxic_conversations_50k
+      name: MTEB ToxicConversationsClassification
+      config: default
+      split: test
+      revision: d7c0de2777da35d6aae2200a62c6e0e5af397c4c
+    metrics:
+    - type: accuracy
+      value: 72.80239999999999
+    - type: ap
+      value: 15.497911013214791
+    - type: f1
+      value: 56.258411577947285
+  - task:
+      type: Classification
+    dataset:
+      type: mteb/tweet_sentiment_extraction
+      name: MTEB TweetSentimentExtractionClassification
+      config: default
+      split: test
+      revision: d604517c81ca91fe16a244d1248fc021f9ecee7a
+    metrics:
+    - type: accuracy
+      value: 61.00452744765139
+    - type: f1
+      value: 61.42228624410908
+  - task:
+      type: Clustering
+    dataset:
+      type: mteb/twentynewsgroups-clustering
+      name: MTEB TwentyNewsgroupsClustering
+      config: default
+      split: test
+      revision: 6125ec4e24fa026cec8a478383ee943acfbd5449
+    metrics:
+    - type: v_measure
+      value: 50.00516915962345
+  - task:
+      type: PairClassification
+    dataset:
+      type: mteb/twittersemeval2015-pairclassification
+      name: MTEB TwitterSemEval2015
+      config: default
+      split: test
+      revision: 70970daeab8776df92f5ea462b6173c0b46fd2d1
+    metrics:
+    - type: cos_sim_accuracy
+      value: 85.62317458425225
+    - type: cos_sim_ap
+      value: 72.95115658063823
+    - type: cos_sim_f1
+      value: 66.78976523344764
+    - type: cos_sim_precision
+      value: 66.77215189873418
+    - type: cos_sim_recall
+      value: 66.80738786279683
+    - type: dot_accuracy
+      value: 85.62317458425225
+    - type: dot_ap
+      value: 73.10385271517778
+    - type: dot_f1
+      value: 66.94853829427399
+    - type: dot_precision
+      value: 61.74242424242424
+    - type: dot_recall
+      value: 73.11345646437995
+    - type: euclidean_accuracy
+      value: 85.65893783155511
+    - type: euclidean_ap
+      value: 72.87428208473992
+    - type: euclidean_f1
+      value: 66.70919994896005
+    - type: euclidean_precision
+      value: 64.5910551025451
+    - type: euclidean_recall
+      value: 68.97097625329816
+    - type: manhattan_accuracy
+      value: 85.59933241938367
+    - type: manhattan_ap
+      value: 72.67282695064966
+    - type: manhattan_f1
+      value: 66.67537215983286
+    - type: manhattan_precision
+      value: 66.00310237849017
+    - type: manhattan_recall
+      value: 67.36147757255937
+    - type: max_accuracy
+      value: 85.65893783155511
+    - type: max_ap
+      value: 73.10385271517778
+    - type: max_f1
+      value: 66.94853829427399
+  - task:
+      type: PairClassification
+    dataset:
+      type: mteb/twitterurlcorpus-pairclassification
+      name: MTEB TwitterURLCorpus
+      config: default
+      split: test
+      revision: 8b6510b0b1fa4e4c4f879467980e9be563ec1cdf
+    metrics:
+    - type: cos_sim_accuracy
+      value: 88.69096130709822
+    - type: cos_sim_ap
+      value: 85.30326978668063
+    - type: cos_sim_f1
+      value: 77.747088683189
+    - type: cos_sim_precision
+      value: 75.4491451753115
+    - type: cos_sim_recall
+      value: 80.189405605174
+    - type: dot_accuracy
+      value: 88.43870066363954
+    - type: dot_ap
+      value: 84.62999949222983
+    - type: dot_f1
+      value: 77.3074661963551
+    - type: dot_precision
+      value: 73.93871239808828
+    - type: dot_recall
+      value: 80.99784416384355
+    - type: euclidean_accuracy
+      value: 88.70066363953894
+    - type: euclidean_ap
+      value: 85.34184508966621
+    - type: euclidean_f1
+      value: 77.76871756856931
+    - type: euclidean_precision
+      value: 74.97855917667239
+    - type: euclidean_recall
+      value: 80.77456113335386
+    - type: manhattan_accuracy
+      value: 88.68319944114566
+    - type: manhattan_ap
+      value: 85.3026464242333
+    - type: manhattan_f1
+      value: 77.66561049296294
+    - type: manhattan_precision
+      value: 74.4665818849795
+    - type: manhattan_recall
+      value: 81.15183246073299
+    - type: max_accuracy
+      value: 88.70066363953894
+    - type: max_ap
+      value: 85.34184508966621
+    - type: max_f1
+      value: 77.76871756856931
+---
+<h1 align="center">GIST small Embedding v0</h1>
+
+*GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning*
+
+The model is fine-tuned on top of the [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5) using the [MEDI dataset](https://github.com/xlang-ai/instructor-embedding.git) augmented with mined triplets from the [MTEB Classification](https://huggingface.co/mteb) training dataset (excluding data from the Amazon Polarity Classification task).
+
+The model does not require any instruction for generating embeddings. This means that queries for retrieval tasks can be directly encoded without crafting instructions.
+
+Technical paper: [GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning](https://arxiv.org/abs/2402.16829)
+
+
+# Data
+
+The dataset used is a compilation of the MEDI and MTEB Classification training datasets. Third-party datasets may be subject to additional terms and conditions under their associated licenses. A HuggingFace Dataset version of the compiled dataset, and the specific revision used to train the model, is available:
+
+- Dataset: [avsolatorio/medi-data-mteb_avs_triplets](https://huggingface.co/datasets/avsolatorio/medi-data-mteb_avs_triplets)
+- Revision: 238a0499b6e6b690cc64ea56fde8461daa8341bb
+
+The dataset contains a `task_type` key, which can be used to select only the mteb classification tasks (prefixed with `mteb_`).
+
+The **MEDI Dataset** is published in the following paper: [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://arxiv.org/abs/2212.09741).
+
+The MTEB Benchmark results of the GIST embedding model, compared with the base model, suggest that the fine-tuning dataset has perturbed the model considerably, which resulted in significant improvements in certain tasks while adversely degrading performance in some.
+
+The retrieval performance for the TRECCOVID task is of note. The fine-tuning dataset does not contain significant knowledge about COVID-19, which could have caused the observed performance degradation. We found some evidence, detailed in the paper, that thematic coverage of the fine-tuning data can affect downstream performance.
+
+# Usage
+
+The model can be easily loaded using the Sentence Transformers library.
+
+```Python
+import torch.nn.functional as F
+from sentence_transformers import SentenceTransformer
+
+revision = None  # Replace with the specific revision to ensure reproducibility if the model is updated.
+
+model = SentenceTransformer("avsolatorio/GIST-small-Embedding-v0", revision=revision)
+
+texts = [
+    "Illustration of the REaLTabFormer model. The left block shows the non-relational tabular data model using GPT-2 with a causal LM head. In contrast, the right block shows how a relational dataset's child table is modeled using a sequence-to-sequence (Seq2Seq) model. The Seq2Seq model uses the observations in the parent table to condition the generation of the observations in the child table. The trained GPT-2 model on the parent table, with weights frozen, is also used as the encoder in the Seq2Seq model.",
+    "Predicting human mobility holds significant practical value, with applications ranging from enhancing disaster risk planning to simulating epidemic spread. In this paper, we present the GeoFormer, a decoder-only transformer model adapted from the GPT architecture to forecast human mobility.",
+    "As the economies of Southeast Asia continue adopting digital technologies, policy makers increasingly ask how to prepare the workforce for emerging labor demands. However, little is known about the skills that workers need to adapt to these changes"
+]
+
+# Compute embeddings
+embeddings = model.encode(texts, convert_to_tensor=True)
+
+# Compute cosine-similarity for each pair of sentences
+scores = F.cosine_similarity(embeddings.unsqueeze(1), embeddings.unsqueeze(0), dim=-1)
+
+print(scores.cpu().numpy())
+```
+
+# Training Parameters
+
+Below are the training parameters used to fine-tune the model:
+
+```
+Epochs = 40
+Warmup ratio = 0.1
+Learning rate = 5e-6
+Batch size = 16
+Checkpoint step = 102000
+Contrastive loss temperature = 0.01
+```
+
+
+# Evaluation
+
+The model was evaluated using the [MTEB Evaluation](https://huggingface.co/mteb) suite.
+
+
+# Citation
+
+Please cite our work if you use GISTEmbed or the datasets we published in your projects or research. 🤗
+
+```
+@article{solatorio2024gistembed,
+    title={GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning},
+    author={Aivin V. Solatorio},
+    journal={arXiv preprint arXiv:2402.16829},
+    year={2024},
+    URL={https://arxiv.org/abs/2402.16829}
+    eprint={2402.16829},
+    archivePrefix={arXiv},
+    primaryClass={cs.LG}
+}
+```
+
+# Acknowledgements
+
+This work is supported by the "KCP IV - Exploring Data Use in the Development Economics Literature using Large Language Models (AI and LLMs)" project funded by the [Knowledge for Change Program (KCP)](https://www.worldbank.org/en/programs/knowledge-for-change) of the World Bank - RA-P503405-RESE-TF0C3444.
+
+The findings, interpretations, and conclusions expressed in this material are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent.

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3b9f58b0c4f047198bae295a3651e2932c2d5aa40821885b8f94ef61d0c09e33
-size 535886480
+oid sha256:0d6fc952cbf4d99d7e5dcb6573ea726372aa460d2dde076141e1682728e7cd6f
+size 578289664

omni_components/embedder.py

@@ -30,7 +30,7 @@ class MultimodalEmbedder(nn.Module):
     - Dimension truncation (MRL)
     """
 
-    def __init__(self, component_dir, legacy=False, max_text_length=128):
+    def __init__(self, component_dir, legacy=False, max_text_length=128, text_pooling="mean"):
         super().__init__()
         root = Path(component_dir)
         self.output_dim = 384
@@ -40,12 +40,15 @@ class MultimodalEmbedder(nn.Module):
         self.num_fusion_layers = 2
         if type(max_text_length) is not int or max_text_length not in (128, 512):
             raise ValueError("Nano max_text_length must be 128 or 512")
+        if text_pooling not in ("mean", "cls"):
+            raise ValueError("Unsupported Nano text pooling")
         self.max_text_length = max_text_length
         self.text_encoder_revision = None
         self.image_encoder_revision = None
         self.audio_encoder_revision = None
         self.text_encoder = TextEncoder(root / "text", output_dim=384, normalize=False,
-                                       max_length=self.max_text_length, enable_layer_outputs=True)
+                                       max_length=self.max_text_length, enable_layer_outputs=True,
+                                       pooling_mode=text_pooling)
         self.image_encoder = ImageEncoder(root / "image", output_dim=384, normalize=False,
                                           enable_layer_outputs=True, legacy=legacy)
         self.audio_encoder = AudioEncoder(root / "audio", output_dim=384, normalize=False,

vela_omni.py

@@ -14,13 +14,16 @@ from omni_components.mini import MultiModalSentenceEmbedder
 
 
 class VelaOmni(nn.Module):
-    def __init__(self, root, variant, legacy=False, audio_projection="identity", text_projection="identity", audio_pooling="mean", max_text_length=None):
+    def __init__(self, root, variant, legacy=False, audio_projection="identity", text_projection="identity", audio_pooling="mean", max_text_length=None, text_pooling="mean"):
         super().__init__()
         self.variant = variant
         if text_projection not in ("identity", "residual_rank64"):
             raise ValueError("Unsupported text projection")
         if audio_pooling not in ("mean", "duration_mean") or (variant != "nano" and audio_pooling != "mean"):
             raise ValueError("Unsupported audio pooling")
+        if text_pooling not in ("mean", "cls") or (variant != "nano" and text_pooling != "mean"):
+            raise ValueError("Unsupported text pooling")
+        self.text_pooling = text_pooling
         self.audio_pooling = audio_pooling
         self.embedding_dim = 384 if variant == "nano" else 768
         if variant not in ("nano", "mini"):
@@ -33,7 +36,7 @@ class VelaOmni(nn.Module):
         self.max_text_length = max_text_length
         if variant == "nano":
             self.model = MultimodalEmbedder(Path(root) / "components", legacy=legacy,
-                                            max_text_length=self.max_text_length)
+                                            max_text_length=self.max_text_length, text_pooling=self.text_pooling)
             self.tokenizer = self.model.text_encoder.tokenizer
             if text_projection == "residual_rank64":
                 self.model.text_encoder.projection = ResidualTextProjection(384)
@@ -59,12 +62,15 @@ class VelaOmni(nn.Module):
             root = Path(snapshot_download(repo_id, revision=revision,
                     allow_patterns=["config.json", "model.safetensors", "components/*"]))
         config = json.loads((root / "config.json").read_text())
-        if config.get("format_version") not in (1, 2, 3):
+        if config.get("format_version") not in (1, 2, 3, 4):
             raise ValueError("Unsupported Vela Omni artifact format")
+        if config.get("image_pooling", "mean") != "mean":
+            raise ValueError("Unsupported image pooling")
         instance = cls(root, config["variant"], audio_projection=config.get("audio_projection", "identity"),
                        text_projection=config.get("text_projection", "identity"),
                        audio_pooling=config.get("audio_pooling", "mean"),
-                       max_text_length=config.get("max_text_length"))
+                       max_text_length=config.get("max_text_length"),
+                       text_pooling=config.get("text_pooling", "mean"))
         weights = load_file(str(root / "model.safetensors"), device="cpu")
         result = instance.model.load_state_dict(weights, strict=True, assign=True)
         if result.missing_keys or result.unexpected_keys: