Reduce Omni Nano size with equivalent embeddings
Browse files- README.md +6 -4
- assets/pareto-arxiv.png +2 -2
- assets/pareto-arxiv.svg +493 -493
- assets/pareto-nmsqa.png +2 -2
- assets/pareto-nmsqa.svg +197 -197
- assets/pareto-sibfleurs.png +2 -2
- assets/pareto-sibfleurs.svg +203 -203
- assets/pareto-vehiclesoundclustering.png +2 -2
- assets/pareto-vehiclesoundclustering.svg +206 -206
- benchmarks/EVALUATION.md +7 -3
- benchmarks/component-equivalence.json +8 -1
- benchmarks/inference-equivalence.json +453 -0
- benchmarks/maeb-audio-only.json +7 -0
- benchmarks/mteb-eng-v2.json +8 -1
- benchmarks/net-progress.json +8 -1
- benchmarks/pareto-data.json +36 -24
- benchmarks/pareto-methodology.md +1 -1
- benchmarks/pareto-points.csv +4 -4
- config.json +11 -10
- model.safetensors +2 -2
- omni_components/single_modality.py +45 -0
- scores.json +57 -12
- vela_omni.py +14 -4
README.md
CHANGED
|
@@ -31,18 +31,20 @@ Text, images and speech in one embedding space. Vela Omni Nano supports multimod
|
|
| 31 |
| Feature | Value |
|
| 32 |
| --- | --- |
|
| 33 |
| Modalities | Text, images, speech |
|
| 34 |
-
| Total parameters |
|
| 35 |
| Embedding dimensions | 384 |
|
| 36 |
| Text context | 512 tokens, including special tokens |
|
| 37 |
| Audio input | Mono, 16 kHz, up to 30 seconds |
|
| 38 |
| Output | L2-normalized vectors; cosine similarity |
|
| 39 |
| License | Apache 2.0; see component attribution |
|
| 40 |
|
|
|
|
|
|
|
| 41 |
## Evaluation
|
| 42 |
|
| 43 |
-
MTEB 2.21.0; scores are 0–100. **Bold marks gains over the [
|
| 44 |
|
| 45 |
-
| Benchmark / metric |
|
| 46 |
| --- | ---: | ---: |
|
| 47 |
| English v2 · 41/41 · mean task | 53.37 | **64.88** |
|
| 48 |
| English v2 · mean task type | 51.98 | **60.78** |
|
|
@@ -53,7 +55,7 @@ Text and speech–text retrieval improve; image classification and clustering ar
|
|
| 53 |
|
| 54 |
### Size and quality
|
| 55 |
|
| 56 |
-
Nano scores **64.89 ArXiv V-measure**, **63.28 NMSQA max AP** and **13.80 Vehicle V-measure**, at
|
| 57 |
|
| 58 |
<table>
|
| 59 |
<tr>
|
|
|
|
| 31 |
| Feature | Value |
|
| 32 |
| --- | --- |
|
| 33 |
| Modalities | Text, images, speech |
|
| 34 |
+
| Total parameters | 135.4M (135,383,808) |
|
| 35 |
| Embedding dimensions | 384 |
|
| 36 |
| Text context | 512 tokens, including special tokens |
|
| 37 |
| Audio input | Mono, 16 kHz, up to 30 seconds |
|
| 38 |
| Output | L2-normalized vectors; cosine similarity |
|
| 39 |
| License | Apache 2.0; see component attribution |
|
| 40 |
|
| 41 |
+
The smaller inference package preserves the encoding computation and all reported scores. [Equivalence and storage details](benchmarks/inference-equivalence.json).
|
| 42 |
+
|
| 43 |
## Evaluation
|
| 44 |
|
| 45 |
+
MTEB 2.21.0; scores are 0–100. **Bold marks gains over the [reference Nano](https://huggingface.co/llm-semantic-router/Vela-1.0-Omni-Nano/blob/3a1efc48995e2b71622d65c6cf71cfc00bbf3aa3/README.md).** Both full panels retain every task and split.
|
| 46 |
|
| 47 |
+
| Benchmark / metric | Reference Nano | Current Nano |
|
| 48 |
| --- | ---: | ---: |
|
| 49 |
| English v2 · 41/41 · mean task | 53.37 | **64.88** |
|
| 50 |
| English v2 · mean task type | 51.98 | **60.78** |
|
|
|
|
| 55 |
|
| 56 |
### Size and quality
|
| 57 |
|
| 58 |
+
Nano scores **64.89 ArXiv V-measure**, **63.28 NMSQA max AP** and **13.80 Vehicle V-measure**, at 135.4M total parameters.
|
| 59 |
|
| 60 |
<table>
|
| 61 |
<tr>
|
assets/pareto-arxiv.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/pareto-arxiv.svg
CHANGED
|
|
|
|
assets/pareto-nmsqa.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/pareto-nmsqa.svg
CHANGED
|
|
|
|
assets/pareto-sibfleurs.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/pareto-sibfleurs.svg
CHANGED
|
|
|
|
assets/pareto-vehiclesoundclustering.png
CHANGED
|
Git LFS Details
|
|
Git LFS Details
|
assets/pareto-vehiclesoundclustering.svg
CHANGED
|
|
|
|
benchmarks/EVALUATION.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
# Evaluation details
|
| 2 |
|
| 3 |
-
This report
|
| 4 |
|
| 5 |
## Complete panels
|
| 6 |
|
|
@@ -11,7 +11,7 @@ English contains all 41 tasks and seven task types. Audio contains all 19 audio-
|
|
| 11 |
| English v2 · 41/41 | 53.37 | **64.88** | 51.98 | **60.78** |
|
| 12 |
| MAEB audio-only · 19/19 | 39.78 | 39.77 | 46.97 | 46.97 |
|
| 13 |
|
| 14 |
-
English was evaluated on an earlier native artifact with exactly the same 199 GIST text tensors and identical source/configuration/tokenizer bytes. Its actual evaluated identity is retained in [the full report](mteb-eng-v2.json) and [text equivalence evidence](component-equivalence.json).
|
| 15 |
|
| 16 |
### All English tasks
|
| 17 |
|
|
@@ -148,8 +148,12 @@ Regressions remain visible: all three image-to-text recall ranks decline; Pets z
|
|
| 148 |
|
| 149 |
The GIST foundation card discloses MTEB classification training data. We do not claim absence of pretraining overlap. Cross-modal optimization uses COCO and LibriSpeech TRAIN only with complete positives and no benchmark/Studio-example training. One final 300-step pair of orthogonal media rotations was exported; text and media encoders are frozen.
|
| 150 |
|
| 151 |
-
The
|
| 152 |
|
| 153 |
## Task-level Pareto comparisons
|
| 154 |
|
| 155 |
[Every plotted observation](pareto-data.json) and [methodology](pareto-methodology.md) retain the reported API snapshot and separate independent peer reruns. F2 reaches 69.7422 ArXiv V-measure and AST reaches 14.3429 Vehicle V-measure. Nano is not on either updated frontier. Parameter counts refer to whole stored models, including all Vela modalities. These four tasks do not establish overall size SOTA.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
# Evaluation details
|
| 2 |
|
| 3 |
+
This report retains measurements from Nano artifact `96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4` (144,554,112 parameters). The previous Nano is [3a1efc48](https://huggingface.co/llm-semantic-router/Vela-1.0-Omni-Nano/tree/3a1efc48995e2b71622d65c6cf71cfc00bbf3aa3) (`fb9cd39c9980a085b75456c715c2295b997784fe733314be68c13635f6384edf`). Scores are shown on a 0–100 scale. Bold means an unrounded improvement over the explicitly named comparator.
|
| 4 |
|
| 5 |
## Complete panels
|
| 6 |
|
|
|
|
| 11 |
| English v2 · 41/41 | 53.37 | **64.88** | 51.98 | **60.78** |
|
| 12 |
| MAEB audio-only · 19/19 | 39.78 | 39.77 | 46.97 | 46.97 |
|
| 13 |
|
| 14 |
+
English was evaluated on an earlier native artifact with exactly the same 199 GIST text tensors and identical source/configuration/tokenizer bytes. Its actual evaluated identity is retained in [the full report](mteb-eng-v2.json) and [text equivalence evidence](component-equivalence.json). Image, audio and cross-modal scores were measured on the 96d artifact. The 135,383,808-parameter inference package preserves the participating weights and public computation; [the equivalence report](inference-equivalence.json) records the exact identities and CPU/GPU synthetic checks. These are applicability checks, not a new benchmark run.
|
| 15 |
|
| 16 |
### All English tasks
|
| 17 |
|
|
|
|
| 148 |
|
| 149 |
The GIST foundation card discloses MTEB classification training data. We do not claim absence of pretraining overlap. Cross-modal optimization uses COCO and LibriSpeech TRAIN only with complete positives and no benchmark/Studio-example training. One final 300-step pair of orthogonal media rotations was exported; text and media encoders are frozen.
|
| 150 |
|
| 151 |
+
The evaluated artifact has 144,554,112 parameters; the current inference package has 135,383,808 parameters and 384-dimensional FP32 vectors (1,536 raw bytes per vector, excluding index overhead). No current-artifact latency or peak-memory claim is made. Historical measurements in scores.json retain their measured artifact identity.
|
| 152 |
|
| 153 |
## Task-level Pareto comparisons
|
| 154 |
|
| 155 |
[Every plotted observation](pareto-data.json) and [methodology](pareto-methodology.md) retain the reported API snapshot and separate independent peer reruns. F2 reaches 69.7422 ArXiv V-measure and AST reaches 14.3429 Vehicle V-measure. Nano is not on either updated frontier. Parameter counts refer to whole stored models, including all Vela modalities. These four tasks do not establish overall size SOTA.
|
| 156 |
+
|
| 157 |
+
## Inference package update
|
| 158 |
+
|
| 159 |
+
The package removes 95 unused tensors (9,170,304 parameters, 6.3439%) while retaining all 476 tensors used by the public encoding paths unchanged. Safetensors storage falls from 578,289,664 to 541,598,024 bytes. Scores, embeddings dimensions, and context limits are unchanged. No latency or measured peak-memory improvement is claimed. The previous quality comparison above keeps its original comparator and measured identities.
|
benchmarks/component-equivalence.json
CHANGED
|
@@ -26,5 +26,12 @@
|
|
| 26 |
"reports": [
|
| 27 |
"benchmarks/mteb-eng-v2.json"
|
| 28 |
],
|
| 29 |
-
"scope": "The full English report keeps the artifact that was actually evaluated. Only text-only scores apply through this bridge. Image, audio and cross-modal results are freshly evaluated on the current artifact."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
}
|
|
|
|
| 26 |
"reports": [
|
| 27 |
"benchmarks/mteb-eng-v2.json"
|
| 28 |
],
|
| 29 |
+
"scope": "The full English report keeps the artifact that was actually evaluated. Only text-only scores apply through this bridge. Image, audio and cross-modal results are freshly evaluated on the current artifact.",
|
| 30 |
+
"inference_profile_applicability": {
|
| 31 |
+
"revision": "self",
|
| 32 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 33 |
+
"parameters": 135383808,
|
| 34 |
+
"report": "benchmarks/inference-equivalence.json",
|
| 35 |
+
"new_benchmark_run": false
|
| 36 |
+
}
|
| 37 |
}
|
benchmarks/inference-equivalence.json
ADDED
|
@@ -0,0 +1,453 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"schema_version": 1,
|
| 3 |
+
"model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 4 |
+
"scope": "Same supported public encode_text, encode_image, and encode_audio computation; no new benchmark run.",
|
| 5 |
+
"evaluated_parent": {
|
| 6 |
+
"revision": "de02c367bc9aa7a6f7ad3e59fb38962502e021c1",
|
| 7 |
+
"native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 8 |
+
"parameters": 144554112
|
| 9 |
+
},
|
| 10 |
+
"applies_to": {
|
| 11 |
+
"revision": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 12 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 13 |
+
"parameters": 135383808,
|
| 14 |
+
"revision_kind": "native_artifact_fingerprint"
|
| 15 |
+
},
|
| 16 |
+
"removed_unused_tensors": 95,
|
| 17 |
+
"removed_parameters": 9170304,
|
| 18 |
+
"removed_percent": 6.3438555106616406,
|
| 19 |
+
"retained_tensors": 476,
|
| 20 |
+
"retained_values_shapes_and_dtypes_exact": true,
|
| 21 |
+
"removed_tensor_keys": [
|
| 22 |
+
"audio_encoder.layer_norms.0.bias",
|
| 23 |
+
"audio_encoder.layer_norms.0.weight",
|
| 24 |
+
"audio_encoder.layer_norms.1.bias",
|
| 25 |
+
"audio_encoder.layer_norms.1.weight",
|
| 26 |
+
"audio_encoder.layer_norms.2.bias",
|
| 27 |
+
"audio_encoder.layer_norms.2.weight",
|
| 28 |
+
"audio_encoder.layer_norms.3.bias",
|
| 29 |
+
"audio_encoder.layer_norms.3.weight",
|
| 30 |
+
"audio_encoder.layer_projections.0.bias",
|
| 31 |
+
"audio_encoder.layer_projections.0.weight",
|
| 32 |
+
"audio_encoder.layer_projections.1.bias",
|
| 33 |
+
"audio_encoder.layer_projections.1.weight",
|
| 34 |
+
"audio_encoder.layer_projections.2.bias",
|
| 35 |
+
"audio_encoder.layer_projections.2.weight",
|
| 36 |
+
"audio_encoder.layer_projections.3.bias",
|
| 37 |
+
"audio_encoder.layer_projections.3.weight",
|
| 38 |
+
"fusion.fusion_layers.0.ffn.0.bias",
|
| 39 |
+
"fusion.fusion_layers.0.ffn.0.weight",
|
| 40 |
+
"fusion.fusion_layers.0.ffn.3.bias",
|
| 41 |
+
"fusion.fusion_layers.0.ffn.3.weight",
|
| 42 |
+
"fusion.fusion_layers.0.norm1.bias",
|
| 43 |
+
"fusion.fusion_layers.0.norm1.weight",
|
| 44 |
+
"fusion.fusion_layers.0.norm2.bias",
|
| 45 |
+
"fusion.fusion_layers.0.norm2.weight",
|
| 46 |
+
"fusion.fusion_layers.0.self_attention.in_proj_bias",
|
| 47 |
+
"fusion.fusion_layers.0.self_attention.in_proj_weight",
|
| 48 |
+
"fusion.fusion_layers.0.self_attention.out_proj.bias",
|
| 49 |
+
"fusion.fusion_layers.0.self_attention.out_proj.weight",
|
| 50 |
+
"fusion.fusion_layers.1.ffn.0.bias",
|
| 51 |
+
"fusion.fusion_layers.1.ffn.0.weight",
|
| 52 |
+
"fusion.fusion_layers.1.ffn.3.bias",
|
| 53 |
+
"fusion.fusion_layers.1.ffn.3.weight",
|
| 54 |
+
"fusion.fusion_layers.1.norm1.bias",
|
| 55 |
+
"fusion.fusion_layers.1.norm1.weight",
|
| 56 |
+
"fusion.fusion_layers.1.norm2.bias",
|
| 57 |
+
"fusion.fusion_layers.1.norm2.weight",
|
| 58 |
+
"fusion.fusion_layers.1.self_attention.in_proj_bias",
|
| 59 |
+
"fusion.fusion_layers.1.self_attention.in_proj_weight",
|
| 60 |
+
"fusion.fusion_layers.1.self_attention.out_proj.bias",
|
| 61 |
+
"fusion.fusion_layers.1.self_attention.out_proj.weight",
|
| 62 |
+
"fusion.input_projections.audio.projector.0.bias",
|
| 63 |
+
"fusion.input_projections.audio.projector.0.weight",
|
| 64 |
+
"fusion.input_projections.audio.projector.1.bias",
|
| 65 |
+
"fusion.input_projections.audio.projector.1.weight",
|
| 66 |
+
"fusion.input_projections.audio.projector.4.bias",
|
| 67 |
+
"fusion.input_projections.audio.projector.4.weight",
|
| 68 |
+
"fusion.input_projections.image.projector.0.bias",
|
| 69 |
+
"fusion.input_projections.image.projector.0.weight",
|
| 70 |
+
"fusion.input_projections.image.projector.1.bias",
|
| 71 |
+
"fusion.input_projections.image.projector.1.weight",
|
| 72 |
+
"fusion.input_projections.image.projector.4.bias",
|
| 73 |
+
"fusion.input_projections.image.projector.4.weight",
|
| 74 |
+
"fusion.input_projections.text.projector.0.bias",
|
| 75 |
+
"fusion.input_projections.text.projector.0.weight",
|
| 76 |
+
"fusion.input_projections.text.projector.1.bias",
|
| 77 |
+
"fusion.input_projections.text.projector.1.weight",
|
| 78 |
+
"fusion.input_projections.text.projector.4.bias",
|
| 79 |
+
"fusion.input_projections.text.projector.4.weight",
|
| 80 |
+
"fusion.layer_norms.0.bias",
|
| 81 |
+
"fusion.layer_norms.0.weight",
|
| 82 |
+
"fusion.layer_norms.1.bias",
|
| 83 |
+
"fusion.layer_norms.1.weight",
|
| 84 |
+
"fusion.layer_projections.0.bias",
|
| 85 |
+
"fusion.layer_projections.0.weight",
|
| 86 |
+
"fusion.layer_projections.1.bias",
|
| 87 |
+
"fusion.layer_projections.1.weight",
|
| 88 |
+
"fusion.modality_embeddings.weight",
|
| 89 |
+
"fusion.output_projection.bias",
|
| 90 |
+
"fusion.output_projection.weight",
|
| 91 |
+
"image_encoder.layer_projections.0.bias",
|
| 92 |
+
"image_encoder.layer_projections.0.weight",
|
| 93 |
+
"image_encoder.layer_projections.1.bias",
|
| 94 |
+
"image_encoder.layer_projections.1.weight",
|
| 95 |
+
"image_encoder.layer_projections.10.bias",
|
| 96 |
+
"image_encoder.layer_projections.10.weight",
|
| 97 |
+
"image_encoder.layer_projections.11.bias",
|
| 98 |
+
"image_encoder.layer_projections.11.weight",
|
| 99 |
+
"image_encoder.layer_projections.2.bias",
|
| 100 |
+
"image_encoder.layer_projections.2.weight",
|
| 101 |
+
"image_encoder.layer_projections.3.bias",
|
| 102 |
+
"image_encoder.layer_projections.3.weight",
|
| 103 |
+
"image_encoder.layer_projections.4.bias",
|
| 104 |
+
"image_encoder.layer_projections.4.weight",
|
| 105 |
+
"image_encoder.layer_projections.5.bias",
|
| 106 |
+
"image_encoder.layer_projections.5.weight",
|
| 107 |
+
"image_encoder.layer_projections.6.bias",
|
| 108 |
+
"image_encoder.layer_projections.6.weight",
|
| 109 |
+
"image_encoder.layer_projections.7.bias",
|
| 110 |
+
"image_encoder.layer_projections.7.weight",
|
| 111 |
+
"image_encoder.layer_projections.8.bias",
|
| 112 |
+
"image_encoder.layer_projections.8.weight",
|
| 113 |
+
"image_encoder.layer_projections.9.bias",
|
| 114 |
+
"image_encoder.layer_projections.9.weight",
|
| 115 |
+
"text_encoder.encoder.pooler.dense.bias",
|
| 116 |
+
"text_encoder.encoder.pooler.dense.weight"
|
| 117 |
+
],
|
| 118 |
+
"supported_public_api": [
|
| 119 |
+
"encode_text",
|
| 120 |
+
"encode_image",
|
| 121 |
+
"encode_audio"
|
| 122 |
+
],
|
| 123 |
+
"omitted_internal_features": [
|
| 124 |
+
"fusion",
|
| 125 |
+
"intermediate layer exits"
|
| 126 |
+
],
|
| 127 |
+
"quality_evidence": "Original common14, selected standard11, English41 and audio19 scores retain their measured artifact identities. English41 first applies from its original 299 artifact to the 96d parent through exact text identity; this public-inference bridge then applies those scores to the smaller package.",
|
| 128 |
+
"text_context_tokens": 512,
|
| 129 |
+
"embedding_dimensions": 384,
|
| 130 |
+
"CPU_synthetic_parity": {
|
| 131 |
+
"all_retained_tensors_exact": true,
|
| 132 |
+
"candidate_fingerprint": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 133 |
+
"device": "cpu",
|
| 134 |
+
"inputs": "deterministic synthetic fixtures, not benchmark quality evaluation",
|
| 135 |
+
"old_artifact_new_loader": {
|
| 136 |
+
"bitwise": true,
|
| 137 |
+
"maxabs": 0.0,
|
| 138 |
+
"relative_L2": 0.0,
|
| 139 |
+
"shape": [
|
| 140 |
+
2,
|
| 141 |
+
384
|
| 142 |
+
]
|
| 143 |
+
},
|
| 144 |
+
"parameters": 135383808,
|
| 145 |
+
"public_boundaries": {
|
| 146 |
+
"129": {
|
| 147 |
+
"comparison": {
|
| 148 |
+
"bitwise": true,
|
| 149 |
+
"maxabs": 0.0,
|
| 150 |
+
"relative_L2": 0.0,
|
| 151 |
+
"shape": [
|
| 152 |
+
1,
|
| 153 |
+
384
|
| 154 |
+
]
|
| 155 |
+
},
|
| 156 |
+
"tokens": 129
|
| 157 |
+
},
|
| 158 |
+
"512": {
|
| 159 |
+
"comparison": {
|
| 160 |
+
"bitwise": true,
|
| 161 |
+
"maxabs": 0.0,
|
| 162 |
+
"relative_L2": 0.0,
|
| 163 |
+
"shape": [
|
| 164 |
+
1,
|
| 165 |
+
384
|
| 166 |
+
]
|
| 167 |
+
},
|
| 168 |
+
"tokens": 512
|
| 169 |
+
},
|
| 170 |
+
"513_rejected": "Text exceeds 512 tokens; shorten or explicitly chunk it",
|
| 171 |
+
"audio_bad_rate_rejected": "Resample audio to 16000 Hz before encoding",
|
| 172 |
+
"audio_empty_rejected": "Audio must contain finite nonempty mono waveforms",
|
| 173 |
+
"audio_over30_rejected": "Audio inputs must be at most 30 seconds",
|
| 174 |
+
"text_empty_list_rejected": "Provide at least one text string"
|
| 175 |
+
},
|
| 176 |
+
"quality_scores_generated": false,
|
| 177 |
+
"same_loaded_object_removed_branches": {
|
| 178 |
+
"audio5": {
|
| 179 |
+
"bitwise": true,
|
| 180 |
+
"maxabs": 0.0,
|
| 181 |
+
"relative_L2": 0.0,
|
| 182 |
+
"shape": [
|
| 183 |
+
5,
|
| 184 |
+
384
|
| 185 |
+
]
|
| 186 |
+
},
|
| 187 |
+
"image4": {
|
| 188 |
+
"bitwise": true,
|
| 189 |
+
"maxabs": 0.0,
|
| 190 |
+
"relative_L2": 0.0,
|
| 191 |
+
"shape": [
|
| 192 |
+
4,
|
| 193 |
+
384
|
| 194 |
+
]
|
| 195 |
+
},
|
| 196 |
+
"text8": {
|
| 197 |
+
"bitwise": true,
|
| 198 |
+
"maxabs": 0.0,
|
| 199 |
+
"relative_L2": 0.0,
|
| 200 |
+
"shape": [
|
| 201 |
+
8,
|
| 202 |
+
384
|
| 203 |
+
]
|
| 204 |
+
},
|
| 205 |
+
"text_single": {
|
| 206 |
+
"bitwise": true,
|
| 207 |
+
"maxabs": 0.0,
|
| 208 |
+
"relative_L2": 0.0,
|
| 209 |
+
"shape": [
|
| 210 |
+
1,
|
| 211 |
+
384
|
| 212 |
+
]
|
| 213 |
+
}
|
| 214 |
+
},
|
| 215 |
+
"separate_loaded_native": {
|
| 216 |
+
"audio5": {
|
| 217 |
+
"bitwise": true,
|
| 218 |
+
"maxabs": 0.0,
|
| 219 |
+
"relative_L2": 0.0,
|
| 220 |
+
"shape": [
|
| 221 |
+
5,
|
| 222 |
+
384
|
| 223 |
+
]
|
| 224 |
+
},
|
| 225 |
+
"image4": {
|
| 226 |
+
"bitwise": true,
|
| 227 |
+
"maxabs": 0.0,
|
| 228 |
+
"relative_L2": 0.0,
|
| 229 |
+
"shape": [
|
| 230 |
+
4,
|
| 231 |
+
384
|
| 232 |
+
]
|
| 233 |
+
},
|
| 234 |
+
"text8": {
|
| 235 |
+
"bitwise": true,
|
| 236 |
+
"maxabs": 0.0,
|
| 237 |
+
"relative_L2": 0.0,
|
| 238 |
+
"shape": [
|
| 239 |
+
8,
|
| 240 |
+
384
|
| 241 |
+
]
|
| 242 |
+
},
|
| 243 |
+
"text_single": {
|
| 244 |
+
"bitwise": true,
|
| 245 |
+
"maxabs": 0.0,
|
| 246 |
+
"relative_L2": 0.0,
|
| 247 |
+
"shape": [
|
| 248 |
+
1,
|
| 249 |
+
384
|
| 250 |
+
]
|
| 251 |
+
}
|
| 252 |
+
},
|
| 253 |
+
"source_sha256": "28cb093c40cc9b042a9e9873fe1819721c93fc243da14c5ec1c5b654e79de95d",
|
| 254 |
+
"status": "passed",
|
| 255 |
+
"supported_public_api": [
|
| 256 |
+
"encode_text",
|
| 257 |
+
"encode_image",
|
| 258 |
+
"encode_audio"
|
| 259 |
+
],
|
| 260 |
+
"torch": "2.8.0+cpu",
|
| 261 |
+
"training_steps": 0,
|
| 262 |
+
"unsupported_internal_features": [
|
| 263 |
+
"fusion",
|
| 264 |
+
"intermediate layer exits"
|
| 265 |
+
]
|
| 266 |
+
},
|
| 267 |
+
"GPU_synthetic_parity": {
|
| 268 |
+
"all_retained_tensors_exact": true,
|
| 269 |
+
"candidate_fingerprint": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 270 |
+
"device": "cuda",
|
| 271 |
+
"inputs": "deterministic synthetic fixtures, not benchmark quality evaluation",
|
| 272 |
+
"old_artifact_new_loader": {
|
| 273 |
+
"bitwise": true,
|
| 274 |
+
"maxabs": 0.0,
|
| 275 |
+
"relative_L2": 0.0,
|
| 276 |
+
"shape": [
|
| 277 |
+
2,
|
| 278 |
+
384
|
| 279 |
+
]
|
| 280 |
+
},
|
| 281 |
+
"parameters": 135383808,
|
| 282 |
+
"public_boundaries": {
|
| 283 |
+
"129": {
|
| 284 |
+
"comparison": {
|
| 285 |
+
"bitwise": true,
|
| 286 |
+
"maxabs": 0.0,
|
| 287 |
+
"relative_L2": 0.0,
|
| 288 |
+
"shape": [
|
| 289 |
+
1,
|
| 290 |
+
384
|
| 291 |
+
]
|
| 292 |
+
},
|
| 293 |
+
"tokens": 129
|
| 294 |
+
},
|
| 295 |
+
"512": {
|
| 296 |
+
"comparison": {
|
| 297 |
+
"bitwise": true,
|
| 298 |
+
"maxabs": 0.0,
|
| 299 |
+
"relative_L2": 0.0,
|
| 300 |
+
"shape": [
|
| 301 |
+
1,
|
| 302 |
+
384
|
| 303 |
+
]
|
| 304 |
+
},
|
| 305 |
+
"tokens": 512
|
| 306 |
+
},
|
| 307 |
+
"513_rejected": "Text exceeds 512 tokens; shorten or explicitly chunk it",
|
| 308 |
+
"audio_bad_rate_rejected": "Resample audio to 16000 Hz before encoding",
|
| 309 |
+
"audio_empty_rejected": "Audio must contain finite nonempty mono waveforms",
|
| 310 |
+
"audio_over30_rejected": "Audio inputs must be at most 30 seconds",
|
| 311 |
+
"text_empty_list_rejected": "Provide at least one text string"
|
| 312 |
+
},
|
| 313 |
+
"quality_scores_generated": false,
|
| 314 |
+
"same_loaded_object_removed_branches": {
|
| 315 |
+
"audio5": {
|
| 316 |
+
"bitwise": true,
|
| 317 |
+
"maxabs": 0.0,
|
| 318 |
+
"relative_L2": 0.0,
|
| 319 |
+
"shape": [
|
| 320 |
+
5,
|
| 321 |
+
384
|
| 322 |
+
]
|
| 323 |
+
},
|
| 324 |
+
"image4": {
|
| 325 |
+
"bitwise": true,
|
| 326 |
+
"maxabs": 0.0,
|
| 327 |
+
"relative_L2": 0.0,
|
| 328 |
+
"shape": [
|
| 329 |
+
4,
|
| 330 |
+
384
|
| 331 |
+
]
|
| 332 |
+
},
|
| 333 |
+
"text8": {
|
| 334 |
+
"bitwise": true,
|
| 335 |
+
"maxabs": 0.0,
|
| 336 |
+
"relative_L2": 0.0,
|
| 337 |
+
"shape": [
|
| 338 |
+
8,
|
| 339 |
+
384
|
| 340 |
+
]
|
| 341 |
+
},
|
| 342 |
+
"text_single": {
|
| 343 |
+
"bitwise": true,
|
| 344 |
+
"maxabs": 0.0,
|
| 345 |
+
"relative_L2": 0.0,
|
| 346 |
+
"shape": [
|
| 347 |
+
1,
|
| 348 |
+
384
|
| 349 |
+
]
|
| 350 |
+
}
|
| 351 |
+
},
|
| 352 |
+
"separate_loaded_native": {
|
| 353 |
+
"audio5": {
|
| 354 |
+
"bitwise": true,
|
| 355 |
+
"maxabs": 0.0,
|
| 356 |
+
"relative_L2": 0.0,
|
| 357 |
+
"shape": [
|
| 358 |
+
5,
|
| 359 |
+
384
|
| 360 |
+
]
|
| 361 |
+
},
|
| 362 |
+
"image4": {
|
| 363 |
+
"bitwise": true,
|
| 364 |
+
"maxabs": 0.0,
|
| 365 |
+
"relative_L2": 0.0,
|
| 366 |
+
"shape": [
|
| 367 |
+
4,
|
| 368 |
+
384
|
| 369 |
+
]
|
| 370 |
+
},
|
| 371 |
+
"text8": {
|
| 372 |
+
"bitwise": true,
|
| 373 |
+
"maxabs": 0.0,
|
| 374 |
+
"relative_L2": 0.0,
|
| 375 |
+
"shape": [
|
| 376 |
+
8,
|
| 377 |
+
384
|
| 378 |
+
]
|
| 379 |
+
},
|
| 380 |
+
"text_single": {
|
| 381 |
+
"bitwise": true,
|
| 382 |
+
"maxabs": 0.0,
|
| 383 |
+
"relative_L2": 0.0,
|
| 384 |
+
"shape": [
|
| 385 |
+
1,
|
| 386 |
+
384
|
| 387 |
+
]
|
| 388 |
+
}
|
| 389 |
+
},
|
| 390 |
+
"source_sha256": "28cb093c40cc9b042a9e9873fe1819721c93fc243da14c5ec1c5b654e79de95d",
|
| 391 |
+
"status": "passed",
|
| 392 |
+
"supported_public_api": [
|
| 393 |
+
"encode_text",
|
| 394 |
+
"encode_image",
|
| 395 |
+
"encode_audio"
|
| 396 |
+
],
|
| 397 |
+
"torch": "2.12.0+git6bbd260",
|
| 398 |
+
"training_steps": 0,
|
| 399 |
+
"unsupported_internal_features": [
|
| 400 |
+
"fusion",
|
| 401 |
+
"intermediate layer exits"
|
| 402 |
+
]
|
| 403 |
+
},
|
| 404 |
+
"cost": {
|
| 405 |
+
"old_safetensors_bytes": 578289664,
|
| 406 |
+
"new_safetensors_bytes": 541598024,
|
| 407 |
+
"reduction_bytes": 36691640,
|
| 408 |
+
"old_FP32_parameter_bytes": 578216448,
|
| 409 |
+
"new_FP32_parameter_bytes": 541535232,
|
| 410 |
+
"scope": "Parameter storage only; not measured process peak memory or latency. Vector and index dimensions are unchanged."
|
| 411 |
+
},
|
| 412 |
+
"native_artifact_files_sha256": {
|
| 413 |
+
"LICENSE": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
|
| 414 |
+
"NOTICE": "ea26e8a96e7088153afab5ab92d629a12c149d8ecb98d6795d9277b059f6a942",
|
| 415 |
+
"components/audio/config.json": "372c430053183035fb9d2d7079482f10053b6ff7d7367525ada9fdeead8adeac",
|
| 416 |
+
"components/audio/preprocessor_config.json": "9b5cd03a36fbb8a627c64d98a5b5b126ead95a77720723944487311f0110b666",
|
| 417 |
+
"components/image/config.json": "c2b09e8c0a60f3405d3e53897358a8a373bc4286234f30fcc9324d065fc13728",
|
| 418 |
+
"components/image/preprocessor_config.json": "5a0e5062d04603dc0e5eff01440f6bebfc66d396f9ca16e0128b31c5730cafc5",
|
| 419 |
+
"components/text/1_Pooling/config.json": "d1caf60c96f5fba2157c0c26b76d80818fad6cf0b8eb5e73ec372ff9818eba5c",
|
| 420 |
+
"components/text/config.json": "419ef27bf56a60adc670c50610f8099c64b23e4a1912f4ee7e321139518569b2",
|
| 421 |
+
"components/text/config_sentence_transformers.json": "940d5f50db195fa6e5e6a4f122c095f77880de259d74b14a65779ed48bdd7c56",
|
| 422 |
+
"components/text/modules.json": "84e40c8e006c9b1d6c122e02cba9b02458120b5fb0c87b746c41e0207cf642cf",
|
| 423 |
+
"components/text/sentence_bert_config.json": "84e39fda68ccbff05bfa723ae9c0e70e23e2ec373b76e0f8c6e71af72a693cbf",
|
| 424 |
+
"components/text/special_tokens_map.json": "5d5b662e421ea9fac075174bb0688ee0d9431699900b90662acd44b2a350503a",
|
| 425 |
+
"components/text/tokenizer.json": "d241a60d5e8f04cc1b2b3e9ef7a4921b27bf526d9f6050ab90f9267a1f9e5c66",
|
| 426 |
+
"components/text/tokenizer_config.json": "0b29c7bfc889e53b36d9dd3e686dd4300f6525110eaa98c76a5dafceb2029f53",
|
| 427 |
+
"components/text/vocab.txt": "07eced375cec144d27c900241f3e339478dec958f92fddbc551f295c992038a3",
|
| 428 |
+
"config.json": "3be38bb9a59ba29b719aa4885844fd9f353606b2118885e4171507ca65b72c4c",
|
| 429 |
+
"licenses/GIST-MODEL-CARD.md": "e2905b6dc66b39a697cd82f3211eea4a7ce45b17cc720eceae7fdd337b601c3c",
|
| 430 |
+
"model.safetensors": "401759e85cec305aca7ff08b31a1a3efa2c410b24b7bb49f9a81de3aabafa52d",
|
| 431 |
+
"omni_components/__init__.py": "543f72d073d77a71e7f5ad857f26d53abbecbbba1ed094336f5bf730af0bd641",
|
| 432 |
+
"omni_components/audio_encoder.py": "2c1e8e347c3e49a438084cc61f486dda9fde00d9f4b4d7caac828aa9b2f77488",
|
| 433 |
+
"omni_components/audio_io.py": "23392eb0784bd92e4b0176004455025faa0d9d47f6216e12da34079c49765546",
|
| 434 |
+
"omni_components/embedder.py": "ad709fd80bc1850f609443777ddf3327765f3ea330659f5c154f90fa8ab5e8fd",
|
| 435 |
+
"omni_components/fusion.py": "a3f6132f190f1533887a77570f4457a0da4eba698574b90888da477919d5c279",
|
| 436 |
+
"omni_components/image_encoder.py": "40e64a4fde0cb1a1416b764a1a8db47105985e3b979bac7b0aa3bc9d789d0f7a",
|
| 437 |
+
"omni_components/mini.py": "c62439716bcf06a0051a8d9ec7ea41f917816133f425a7fa004561279daac4fe",
|
| 438 |
+
"omni_components/records.py": "acb3a3cda9d260e39007b6aafd74efd002de974700e14a5535f66590b8b801d6",
|
| 439 |
+
"omni_components/residual_projection.py": "30028685698c9b4779c44f8ebe28bf80c1689b99594ca4b287db16d133227ad4",
|
| 440 |
+
"omni_components/single_modality.py": "a61edcdcac3dbad5c6b84101655e290fa5d74d7292bd8f61105772d9b2135171",
|
| 441 |
+
"omni_components/text_backbone.py": "c11a92203f3010ef857d2c809257781150a790acf1877ed83e1a0021449268c3",
|
| 442 |
+
"omni_components/text_encoder.py": "578329b6bf656423e0b9543783f30a6455449a7402fd3a1a9f601a8afc9705af",
|
| 443 |
+
"vela_omni.py": "a04bd0e9b163b0f440128a3086632d0e6223312e592f247f483c221210f08048"
|
| 444 |
+
},
|
| 445 |
+
"evidence_sha256": {
|
| 446 |
+
"BUILD.json": "d3701e04ca5a0949d4da29994a15ea69847c8fbf44fb7ce9757e1ccc1d051373",
|
| 447 |
+
"PARITY_CPU.json": "9d75c24759dc47776a1892b7b9599d2f7a242adb527707ec1054008a1e785540",
|
| 448 |
+
"PARITY_CUDA.json": "99f23ead7e05aea2c30e96361e4d617175fe7290563933c0203db45057938729"
|
| 449 |
+
},
|
| 450 |
+
"training_steps": 0,
|
| 451 |
+
"new_quality_scores": false,
|
| 452 |
+
"new_latency_claim": false
|
| 453 |
+
}
|
benchmarks/maeb-audio-only.json
CHANGED
|
@@ -11852,5 +11852,12 @@
|
|
| 11852 |
"VoxPopuliAccentPairClassification": -0.02844431688273641,
|
| 11853 |
"VoxPopuliGenderClustering": 0.0,
|
| 11854 |
"VoxPopuliLanguageID": 0.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 11855 |
}
|
| 11856 |
}
|
|
|
|
| 11852 |
"VoxPopuliAccentPairClassification": -0.02844431688273641,
|
| 11853 |
"VoxPopuliGenderClustering": 0.0,
|
| 11854 |
"VoxPopuliLanguageID": 0.0
|
| 11855 |
+
},
|
| 11856 |
+
"inference_profile_applicability": {
|
| 11857 |
+
"revision": "self",
|
| 11858 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 11859 |
+
"parameters": 135383808,
|
| 11860 |
+
"report": "benchmarks/inference-equivalence.json",
|
| 11861 |
+
"new_benchmark_run": false
|
| 11862 |
}
|
| 11863 |
}
|
benchmarks/mteb-eng-v2.json
CHANGED
|
@@ -5011,5 +5011,12 @@
|
|
| 5011 |
"task": "STS22.v2",
|
| 5012 |
"task_type": "STS"
|
| 5013 |
}
|
| 5014 |
-
]
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 5015 |
}
|
|
|
|
| 5011 |
"task": "STS22.v2",
|
| 5012 |
"task_type": "STS"
|
| 5013 |
}
|
| 5014 |
+
],
|
| 5015 |
+
"inference_profile_applicability": {
|
| 5016 |
+
"revision": "self",
|
| 5017 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 5018 |
+
"parameters": 135383808,
|
| 5019 |
+
"report": "benchmarks/inference-equivalence.json",
|
| 5020 |
+
"new_benchmark_run": false
|
| 5021 |
+
}
|
| 5022 |
}
|
benchmarks/net-progress.json
CHANGED
|
@@ -536,5 +536,12 @@
|
|
| 536 |
"VoxPopuliGenderClustering": 0.0,
|
| 537 |
"VoxPopuliLanguageID": 0.0
|
| 538 |
},
|
| 539 |
-
"paired_interval_scope": "Paired intervals cover the original common metrics, family and two-primary-per-family utility. They are not confidence intervals for the new all-14 common mean or the four-component product aggregate."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 540 |
}
|
|
|
|
| 536 |
"VoxPopuliGenderClustering": 0.0,
|
| 537 |
"VoxPopuliLanguageID": 0.0
|
| 538 |
},
|
| 539 |
+
"paired_interval_scope": "Paired intervals cover the original common metrics, family and two-primary-per-family utility. They are not confidence intervals for the new all-14 common mean or the four-component product aggregate.",
|
| 540 |
+
"inference_profile_applicability": {
|
| 541 |
+
"revision": "self",
|
| 542 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 543 |
+
"parameters": 135383808,
|
| 544 |
+
"report": "benchmarks/inference-equivalence.json",
|
| 545 |
+
"new_benchmark_run": false
|
| 546 |
+
}
|
| 547 |
}
|
benchmarks/pareto-data.json
CHANGED
|
@@ -2569,8 +2569,8 @@
|
|
| 2569 |
},
|
| 2570 |
{
|
| 2571 |
"model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 2572 |
-
"parameters_billion": 0.
|
| 2573 |
-
"parameters_exact":
|
| 2574 |
"score": 64.88847259326147,
|
| 2575 |
"open_weights": true,
|
| 2576 |
"modalities": [
|
|
@@ -2582,15 +2582,18 @@
|
|
| 2582 |
"kind": "nano",
|
| 2583 |
"evaluated_task_revision": "0bbdb47bcbe3a90093699aefeed338a0f28a7ee8",
|
| 2584 |
"on_displayed_frontier": false,
|
| 2585 |
-
"evidence_type": "
|
| 2586 |
"observation_id": "vela_release_evaluation:llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 2587 |
-
"native_artifact_sha256": "
|
| 2588 |
-
"revision": "
|
| 2589 |
"evaluated_native_artifact_sha256": "29994142677559ca657c27b657eda0dddbf583bd4cf42a3a31095ab3f99d1646",
|
| 2590 |
"evaluated_revision": "29994142677559ca657c27b657eda0dddbf583bd4cf42a3a31095ab3f99d1646",
|
| 2591 |
"evaluated_model": "local/Vela-Omni-Nano-GIST-Rotation",
|
| 2592 |
"revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
|
| 2593 |
-
"report": "benchmarks/mteb-eng-v2.json"
|
|
|
|
|
|
|
|
|
|
| 2594 |
},
|
| 2595 |
{
|
| 2596 |
"model": "ibm-granite/granite-embedding-english-r2",
|
|
@@ -9600,8 +9603,8 @@
|
|
| 9600 |
},
|
| 9601 |
{
|
| 9602 |
"model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 9603 |
-
"parameters_billion": 0.
|
| 9604 |
-
"parameters_exact":
|
| 9605 |
"score": 63.28340215485784,
|
| 9606 |
"open_weights": true,
|
| 9607 |
"modalities": [
|
|
@@ -9613,14 +9616,17 @@
|
|
| 9613 |
"kind": "nano",
|
| 9614 |
"evaluated_task_revision": "d33eea302d8a64c0a2ee094cf39d77c0814fe87a",
|
| 9615 |
"on_displayed_frontier": true,
|
| 9616 |
-
"native_artifact_sha256": "
|
| 9617 |
-
"revision": "
|
| 9618 |
"evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 9619 |
"evaluated_revision": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 9620 |
"evaluated_model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 9621 |
"revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
|
| 9622 |
-
"evidence_type": "
|
| 9623 |
-
"report": "benchmarks/maeb-audio-only.json"
|
|
|
|
|
|
|
|
|
|
| 9624 |
},
|
| 9625 |
{
|
| 9626 |
"model": "laion/clap-htsat-unfused",
|
|
@@ -11233,8 +11239,8 @@
|
|
| 11233 |
},
|
| 11234 |
{
|
| 11235 |
"model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 11236 |
-
"parameters_billion": 0.
|
| 11237 |
-
"parameters_exact":
|
| 11238 |
"score": 18.304657831512046,
|
| 11239 |
"open_weights": true,
|
| 11240 |
"modalities": [
|
|
@@ -11246,14 +11252,17 @@
|
|
| 11246 |
"kind": "nano",
|
| 11247 |
"evaluated_task_revision": "186b61175fbd77059f769b6bf1110d449a2ff311",
|
| 11248 |
"on_displayed_frontier": false,
|
| 11249 |
-
"native_artifact_sha256": "
|
| 11250 |
-
"revision": "
|
| 11251 |
"evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 11252 |
"evaluated_revision": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 11253 |
"evaluated_model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 11254 |
"revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
|
| 11255 |
-
"evidence_type": "
|
| 11256 |
-
"report": "benchmarks/maeb-audio-only.json"
|
|
|
|
|
|
|
|
|
|
| 11257 |
},
|
| 11258 |
{
|
| 11259 |
"model": "laion/clap-htsat-unfused",
|
|
@@ -12912,8 +12921,8 @@
|
|
| 12912 |
},
|
| 12913 |
{
|
| 12914 |
"model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 12915 |
-
"parameters_billion": 0.
|
| 12916 |
-
"parameters_exact":
|
| 12917 |
"score": 13.800701178563807,
|
| 12918 |
"open_weights": true,
|
| 12919 |
"modalities": [
|
|
@@ -12925,14 +12934,17 @@
|
|
| 12925 |
"kind": "nano",
|
| 12926 |
"evaluated_task_revision": "5f905f9544d0c78b607639662c88c2ddd1216384",
|
| 12927 |
"on_displayed_frontier": false,
|
| 12928 |
-
"native_artifact_sha256": "
|
| 12929 |
-
"revision": "
|
| 12930 |
"evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 12931 |
"evaluated_revision": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 12932 |
"evaluated_model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 12933 |
"revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
|
| 12934 |
-
"evidence_type": "
|
| 12935 |
-
"report": "benchmarks/maeb-audio-only.json"
|
|
|
|
|
|
|
|
|
|
| 12936 |
},
|
| 12937 |
{
|
| 12938 |
"model": "laion/clap-htsat-unfused",
|
|
|
|
| 2569 |
},
|
| 2570 |
{
|
| 2571 |
"model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 2572 |
+
"parameters_billion": 0.135383808,
|
| 2573 |
+
"parameters_exact": 135383808,
|
| 2574 |
"score": 64.88847259326147,
|
| 2575 |
"open_weights": true,
|
| 2576 |
"modalities": [
|
|
|
|
| 2582 |
"kind": "nano",
|
| 2583 |
"evaluated_task_revision": "0bbdb47bcbe3a90093699aefeed338a0f28a7ee8",
|
| 2584 |
"on_displayed_frontier": false,
|
| 2585 |
+
"evidence_type": "exact_public_inference_applicability",
|
| 2586 |
"observation_id": "vela_release_evaluation:llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 2587 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 2588 |
+
"revision": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 2589 |
"evaluated_native_artifact_sha256": "29994142677559ca657c27b657eda0dddbf583bd4cf42a3a31095ab3f99d1646",
|
| 2590 |
"evaluated_revision": "29994142677559ca657c27b657eda0dddbf583bd4cf42a3a31095ab3f99d1646",
|
| 2591 |
"evaluated_model": "local/Vela-Omni-Nano-GIST-Rotation",
|
| 2592 |
"revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
|
| 2593 |
+
"report": "benchmarks/mteb-eng-v2.json",
|
| 2594 |
+
"prior_applicable_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 2595 |
+
"inference_equivalence_report": "benchmarks/inference-equivalence.json",
|
| 2596 |
+
"revision_kind": "native_artifact_fingerprint"
|
| 2597 |
},
|
| 2598 |
{
|
| 2599 |
"model": "ibm-granite/granite-embedding-english-r2",
|
|
|
|
| 9603 |
},
|
| 9604 |
{
|
| 9605 |
"model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 9606 |
+
"parameters_billion": 0.135383808,
|
| 9607 |
+
"parameters_exact": 135383808,
|
| 9608 |
"score": 63.28340215485784,
|
| 9609 |
"open_weights": true,
|
| 9610 |
"modalities": [
|
|
|
|
| 9616 |
"kind": "nano",
|
| 9617 |
"evaluated_task_revision": "d33eea302d8a64c0a2ee094cf39d77c0814fe87a",
|
| 9618 |
"on_displayed_frontier": true,
|
| 9619 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 9620 |
+
"revision": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 9621 |
"evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 9622 |
"evaluated_revision": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 9623 |
"evaluated_model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 9624 |
"revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
|
| 9625 |
+
"evidence_type": "exact_public_inference_applicability",
|
| 9626 |
+
"report": "benchmarks/maeb-audio-only.json",
|
| 9627 |
+
"prior_applicable_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 9628 |
+
"inference_equivalence_report": "benchmarks/inference-equivalence.json",
|
| 9629 |
+
"revision_kind": "native_artifact_fingerprint"
|
| 9630 |
},
|
| 9631 |
{
|
| 9632 |
"model": "laion/clap-htsat-unfused",
|
|
|
|
| 11239 |
},
|
| 11240 |
{
|
| 11241 |
"model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 11242 |
+
"parameters_billion": 0.135383808,
|
| 11243 |
+
"parameters_exact": 135383808,
|
| 11244 |
"score": 18.304657831512046,
|
| 11245 |
"open_weights": true,
|
| 11246 |
"modalities": [
|
|
|
|
| 11252 |
"kind": "nano",
|
| 11253 |
"evaluated_task_revision": "186b61175fbd77059f769b6bf1110d449a2ff311",
|
| 11254 |
"on_displayed_frontier": false,
|
| 11255 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 11256 |
+
"revision": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 11257 |
"evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 11258 |
"evaluated_revision": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 11259 |
"evaluated_model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 11260 |
"revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
|
| 11261 |
+
"evidence_type": "exact_public_inference_applicability",
|
| 11262 |
+
"report": "benchmarks/maeb-audio-only.json",
|
| 11263 |
+
"prior_applicable_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 11264 |
+
"inference_equivalence_report": "benchmarks/inference-equivalence.json",
|
| 11265 |
+
"revision_kind": "native_artifact_fingerprint"
|
| 11266 |
},
|
| 11267 |
{
|
| 11268 |
"model": "laion/clap-htsat-unfused",
|
|
|
|
| 12921 |
},
|
| 12922 |
{
|
| 12923 |
"model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 12924 |
+
"parameters_billion": 0.135383808,
|
| 12925 |
+
"parameters_exact": 135383808,
|
| 12926 |
"score": 13.800701178563807,
|
| 12927 |
"open_weights": true,
|
| 12928 |
"modalities": [
|
|
|
|
| 12934 |
"kind": "nano",
|
| 12935 |
"evaluated_task_revision": "5f905f9544d0c78b607639662c88c2ddd1216384",
|
| 12936 |
"on_displayed_frontier": false,
|
| 12937 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 12938 |
+
"revision": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 12939 |
"evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 12940 |
"evaluated_revision": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 12941 |
"evaluated_model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 12942 |
"revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
|
| 12943 |
+
"evidence_type": "exact_public_inference_applicability",
|
| 12944 |
+
"report": "benchmarks/maeb-audio-only.json",
|
| 12945 |
+
"prior_applicable_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 12946 |
+
"inference_equivalence_report": "benchmarks/inference-equivalence.json",
|
| 12947 |
+
"revision_kind": "native_artifact_fingerprint"
|
| 12948 |
},
|
| 12949 |
{
|
| 12950 |
"model": "laion/clap-htsat-unfused",
|
benchmarks/pareto-methodology.md
CHANGED
|
@@ -6,6 +6,6 @@ The frontier uses every displayed observation: a point is dominated if another i
|
|
| 6 |
|
| 7 |
Reported and independently measured observations remain separate, even for the same model. F2's reported size has an explicit checkpoint-based correction in its report; its actual rerun uses the official task instruction, BF16 and an 8192-token limit. AST's old API point has 87M rounded parameters and no revision; it is not silently normalized. The new AST point counts all 86,594,063 parameters of the pinned checkpoint (86,187,264 execute as the encoder) and uses the official AST processor/raw pooler, FP32, batch eight, all 1,705 Vehicle rows and ten default clustering repetitions with seed 20260918. No training or task tuning is involved. The original NumPy/tensor interface failure and the narrow compatible wrapper are retained in its evidence.
|
| 8 |
|
| 9 |
-
Nano's English coordinate retains the actual evaluated artifact and applies through exact text identity.
|
| 10 |
|
| 11 |
[All plotted points (CSV)](pareto-points.csv) · [F2 rerun](peer-evaluations/f2llm-v2-80m-arxiv.json) · [AST rerun](peer-evaluations/ast-vehicle.json) · [Complete evaluation](EVALUATION.md)
|
|
|
|
| 6 |
|
| 7 |
Reported and independently measured observations remain separate, even for the same model. F2's reported size has an explicit checkpoint-based correction in its report; its actual rerun uses the official task instruction, BF16 and an 8192-token limit. AST's old API point has 87M rounded parameters and no revision; it is not silently normalized. The new AST point counts all 86,594,063 parameters of the pinned checkpoint (86,187,264 execute as the encoder) and uses the official AST processor/raw pooler, FP32, batch eight, all 1,705 Vehicle rows and ten default clustering repetitions with seed 20260918. No training or task tuning is involved. The original NumPy/tensor interface failure and the narrow compatible wrapper are retained in its evidence.
|
| 8 |
|
| 9 |
+
Nano's English coordinate retains the actual evaluated artifact and applies through exact text identity. Nano audio coordinates retain their 96d evaluated identity. Nano now has 135,383,808 stored parameters after unused-branch removal; CPU/GPU bitwise synthetic parity and unchanged participating tensors support score applicability through [the public-inference bridge](inference-equivalence.json). No benchmark rerun or score improvement is implied. All raw identities, parameters, metrics and rich results remain in the linked reports. Within-task API protocol differences and unknown training overlap limit causal comparison. No official benchmark-wide or size-SOTA claim follows from these four task-level plots.
|
| 10 |
|
| 11 |
[All plotted points (CSV)](pareto-points.csv) · [F2 rerun](peer-evaluations/f2llm-v2-80m-arxiv.json) · [AST rerun](peer-evaluations/ast-vehicle.json) · [Complete evaluation](EVALUATION.md)
|
benchmarks/pareto-points.csv
CHANGED
|
@@ -81,7 +81,7 @@ ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1,peer,0.137
|
|
| 81 |
ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1-ablated,peer,0.137,57.903400000000005,True,False,registry_reported,registry_reported:nomic-ai/nomic-embed-text-v1-ablated,,
|
| 82 |
ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1-unsupervised,peer,0.137,56.7453,True,False,registry_reported,registry_reported:nomic-ai/nomic-embed-text-v1-unsupervised,,
|
| 83 |
ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1.5,peer,0.137,60.08051520875409,True,False,registry_reported,registry_reported:nomic-ai/nomic-embed-text-v1.5,,
|
| 84 |
-
ArXivHierarchicalClusteringP2P,v_measure,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.
|
| 85 |
ArXivHierarchicalClusteringP2P,v_measure,ibm-granite/granite-embedding-english-r2,peer,0.149,59.0566,True,False,registry_reported,registry_reported:ibm-granite/granite-embedding-english-r2,,
|
| 86 |
ArXivHierarchicalClusteringP2P,v_measure,Tarka-AIR/Tarka-Embedding-150M-V1,peer,0.156,63.397099999999995,True,False,registry_reported,registry_reported:Tarka-AIR/Tarka-Embedding-150M-V1,,
|
| 87 |
ArXivHierarchicalClusteringP2P,v_measure,codefuse-ai/F2LLM-v2-160M,peer,0.159,62.4399,True,False,registry_reported,registry_reported:codefuse-ai/F2LLM-v2-160M,,
|
|
@@ -234,7 +234,7 @@ NMSQAPairClassification,max_ap,microsoft/wavlm-base-plus-sv,peer,0.095,48.4168,T
|
|
| 234 |
NMSQAPairClassification,max_ap,microsoft/wavlm-base-sd,peer,0.095,52.9964,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sd,,
|
| 235 |
NMSQAPairClassification,max_ap,microsoft/wavlm-base-sv,peer,0.095,52.9964,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sv,,
|
| 236 |
NMSQAPairClassification,max_ap,asapp/sew-d-mid-400k-ft-ls100h,peer,0.139,48.4208,True,False,registry_reported,registry_reported:asapp/sew-d-mid-400k-ft-ls100h,,
|
| 237 |
-
NMSQAPairClassification,max_ap,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.
|
| 238 |
NMSQAPairClassification,max_ap,laion/clap-htsat-unfused,peer,0.153,46.9791,True,False,registry_reported,registry_reported:laion/clap-htsat-unfused,,
|
| 239 |
NMSQAPairClassification,max_ap,laion/clap-htsat-fused,peer,0.154,47.8507,True,False,registry_reported,registry_reported:laion/clap-htsat-fused,,
|
| 240 |
NMSQAPairClassification,max_ap,microsoft/msclap-2023,peer,0.16,51.0934,True,False,registry_reported,registry_reported:microsoft/msclap-2023,,
|
|
@@ -302,7 +302,7 @@ SIBFLEURS,accuracy,microsoft/wavlm-base-plus-sv,peer,0.095,11.857322549019608,Tr
|
|
| 302 |
SIBFLEURS,accuracy,microsoft/wavlm-base-sd,peer,0.095,13.31318431372549,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sd,,
|
| 303 |
SIBFLEURS,accuracy,microsoft/wavlm-base-sv,peer,0.095,13.31318431372549,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sv,,
|
| 304 |
SIBFLEURS,accuracy,asapp/sew-d-mid-400k-ft-ls100h,peer,0.139,12.634273529411763,True,False,registry_reported,registry_reported:asapp/sew-d-mid-400k-ft-ls100h,,
|
| 305 |
-
SIBFLEURS,accuracy,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.
|
| 306 |
SIBFLEURS,accuracy,laion/clap-htsat-unfused,peer,0.153,8.674730392156862,True,False,registry_reported,registry_reported:laion/clap-htsat-unfused,,
|
| 307 |
SIBFLEURS,accuracy,laion/clap-htsat-fused,peer,0.154,9.069597058823529,True,False,registry_reported,registry_reported:laion/clap-htsat-fused,,
|
| 308 |
SIBFLEURS,accuracy,microsoft/msclap-2023,peer,0.16,14.073474509803921,True,False,registry_reported,registry_reported:microsoft/msclap-2023,,
|
|
@@ -372,7 +372,7 @@ VehicleSoundClustering,v_measure,microsoft/wavlm-base-plus-sv,peer,0.095,1.7932,
|
|
| 372 |
VehicleSoundClustering,v_measure,microsoft/wavlm-base-sd,peer,0.095,0.40940000000000004,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sd,,
|
| 373 |
VehicleSoundClustering,v_measure,microsoft/wavlm-base-sv,peer,0.095,0.40940000000000004,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sv,,
|
| 374 |
VehicleSoundClustering,v_measure,asapp/sew-d-mid-400k-ft-ls100h,peer,0.139,0.6276,True,False,registry_reported,registry_reported:asapp/sew-d-mid-400k-ft-ls100h,,
|
| 375 |
-
VehicleSoundClustering,v_measure,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.
|
| 376 |
VehicleSoundClustering,v_measure,laion/clap-htsat-unfused,peer,0.153,2.6642,True,False,registry_reported,registry_reported:laion/clap-htsat-unfused,,
|
| 377 |
VehicleSoundClustering,v_measure,laion/clap-htsat-fused,peer,0.154,4.7132,True,False,registry_reported,registry_reported:laion/clap-htsat-fused,,
|
| 378 |
VehicleSoundClustering,v_measure,microsoft/msclap-2023,peer,0.16,2.86,True,False,registry_reported,registry_reported:microsoft/msclap-2023,,
|
|
|
|
| 81 |
ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1-ablated,peer,0.137,57.903400000000005,True,False,registry_reported,registry_reported:nomic-ai/nomic-embed-text-v1-ablated,,
|
| 82 |
ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1-unsupervised,peer,0.137,56.7453,True,False,registry_reported,registry_reported:nomic-ai/nomic-embed-text-v1-unsupervised,,
|
| 83 |
ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1.5,peer,0.137,60.08051520875409,True,False,registry_reported,registry_reported:nomic-ai/nomic-embed-text-v1.5,,
|
| 84 |
+
ArXivHierarchicalClusteringP2P,v_measure,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.135383808,64.88847259326147,True,False,exact_public_inference_applicability,vela_release_evaluation:llm-semantic-router/Vela-1.0-Omni-Nano,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa
|
| 85 |
ArXivHierarchicalClusteringP2P,v_measure,ibm-granite/granite-embedding-english-r2,peer,0.149,59.0566,True,False,registry_reported,registry_reported:ibm-granite/granite-embedding-english-r2,,
|
| 86 |
ArXivHierarchicalClusteringP2P,v_measure,Tarka-AIR/Tarka-Embedding-150M-V1,peer,0.156,63.397099999999995,True,False,registry_reported,registry_reported:Tarka-AIR/Tarka-Embedding-150M-V1,,
|
| 87 |
ArXivHierarchicalClusteringP2P,v_measure,codefuse-ai/F2LLM-v2-160M,peer,0.159,62.4399,True,False,registry_reported,registry_reported:codefuse-ai/F2LLM-v2-160M,,
|
|
|
|
| 234 |
NMSQAPairClassification,max_ap,microsoft/wavlm-base-sd,peer,0.095,52.9964,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sd,,
|
| 235 |
NMSQAPairClassification,max_ap,microsoft/wavlm-base-sv,peer,0.095,52.9964,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sv,,
|
| 236 |
NMSQAPairClassification,max_ap,asapp/sew-d-mid-400k-ft-ls100h,peer,0.139,48.4208,True,False,registry_reported,registry_reported:asapp/sew-d-mid-400k-ft-ls100h,,
|
| 237 |
+
NMSQAPairClassification,max_ap,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.135383808,63.28340215485784,True,True,exact_public_inference_applicability,,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa
|
| 238 |
NMSQAPairClassification,max_ap,laion/clap-htsat-unfused,peer,0.153,46.9791,True,False,registry_reported,registry_reported:laion/clap-htsat-unfused,,
|
| 239 |
NMSQAPairClassification,max_ap,laion/clap-htsat-fused,peer,0.154,47.8507,True,False,registry_reported,registry_reported:laion/clap-htsat-fused,,
|
| 240 |
NMSQAPairClassification,max_ap,microsoft/msclap-2023,peer,0.16,51.0934,True,False,registry_reported,registry_reported:microsoft/msclap-2023,,
|
|
|
|
| 302 |
SIBFLEURS,accuracy,microsoft/wavlm-base-sd,peer,0.095,13.31318431372549,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sd,,
|
| 303 |
SIBFLEURS,accuracy,microsoft/wavlm-base-sv,peer,0.095,13.31318431372549,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sv,,
|
| 304 |
SIBFLEURS,accuracy,asapp/sew-d-mid-400k-ft-ls100h,peer,0.139,12.634273529411763,True,False,registry_reported,registry_reported:asapp/sew-d-mid-400k-ft-ls100h,,
|
| 305 |
+
SIBFLEURS,accuracy,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.135383808,18.304657831512046,True,False,exact_public_inference_applicability,,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa
|
| 306 |
SIBFLEURS,accuracy,laion/clap-htsat-unfused,peer,0.153,8.674730392156862,True,False,registry_reported,registry_reported:laion/clap-htsat-unfused,,
|
| 307 |
SIBFLEURS,accuracy,laion/clap-htsat-fused,peer,0.154,9.069597058823529,True,False,registry_reported,registry_reported:laion/clap-htsat-fused,,
|
| 308 |
SIBFLEURS,accuracy,microsoft/msclap-2023,peer,0.16,14.073474509803921,True,False,registry_reported,registry_reported:microsoft/msclap-2023,,
|
|
|
|
| 372 |
VehicleSoundClustering,v_measure,microsoft/wavlm-base-sd,peer,0.095,0.40940000000000004,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sd,,
|
| 373 |
VehicleSoundClustering,v_measure,microsoft/wavlm-base-sv,peer,0.095,0.40940000000000004,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sv,,
|
| 374 |
VehicleSoundClustering,v_measure,asapp/sew-d-mid-400k-ft-ls100h,peer,0.139,0.6276,True,False,registry_reported,registry_reported:asapp/sew-d-mid-400k-ft-ls100h,,
|
| 375 |
+
VehicleSoundClustering,v_measure,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.135383808,13.800701178563807,True,False,exact_public_inference_applicability,,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa
|
| 376 |
VehicleSoundClustering,v_measure,laion/clap-htsat-unfused,peer,0.153,2.6642,True,False,registry_reported,registry_reported:laion/clap-htsat-unfused,,
|
| 377 |
VehicleSoundClustering,v_measure,laion/clap-htsat-fused,peer,0.154,4.7132,True,False,registry_reported,registry_reported:laion/clap-htsat-fused,,
|
| 378 |
VehicleSoundClustering,v_measure,microsoft/msclap-2023,peer,0.16,2.86,True,False,registry_reported,registry_reported:microsoft/msclap-2023,,
|
config.json
CHANGED
|
@@ -1,18 +1,19 @@
|
|
| 1 |
{
|
| 2 |
-
"format_version": 4,
|
| 3 |
-
"variant": "nano",
|
| 4 |
-
"parameter_count": 144554112,
|
| 5 |
-
"embedding_dim": 384,
|
| 6 |
-
"max_text_length": 512,
|
| 7 |
"architectures": [
|
| 8 |
"VelaOmni"
|
| 9 |
],
|
| 10 |
-
"torch_dtype": "float32",
|
| 11 |
-
"library_name": "pytorch",
|
| 12 |
-
"audio_sampling_rate": 16000,
|
| 13 |
"audio_max_seconds": 30,
|
|
|
|
| 14 |
"audio_projection": "linear",
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 15 |
"text_projection": "identity",
|
| 16 |
-
"
|
| 17 |
-
"
|
| 18 |
}
|
|
|
|
| 1 |
{
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2 |
"architectures": [
|
| 3 |
"VelaOmni"
|
| 4 |
],
|
|
|
|
|
|
|
|
|
|
| 5 |
"audio_max_seconds": 30,
|
| 6 |
+
"audio_pooling": "mean",
|
| 7 |
"audio_projection": "linear",
|
| 8 |
+
"audio_sampling_rate": 16000,
|
| 9 |
+
"embedding_dim": 384,
|
| 10 |
+
"format_version": 5,
|
| 11 |
+
"inference_profile": "single_modality",
|
| 12 |
+
"library_name": "pytorch",
|
| 13 |
+
"max_text_length": 512,
|
| 14 |
+
"parameter_count": 135383808,
|
| 15 |
+
"text_pooling": "cls",
|
| 16 |
"text_projection": "identity",
|
| 17 |
+
"torch_dtype": "float32",
|
| 18 |
+
"variant": "nano"
|
| 19 |
}
|
model.safetensors
CHANGED
|
@@ -1,3 +1,3 @@
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
-
oid sha256:
|
| 3 |
-
size
|
|
|
|
| 1 |
version https://git-lfs.github.com/spec/v1
|
| 2 |
+
oid sha256:401759e85cec305aca7ff08b31a1a3efa2c410b24b7bb49f9a81de3aabafa52d
|
| 3 |
+
size 541598024
|
omni_components/single_modality.py
ADDED
|
@@ -0,0 +1,45 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
"""Nano inference runtime for the three public modality encoders."""
|
| 2 |
+
from pathlib import Path
|
| 3 |
+
from torch import nn
|
| 4 |
+
|
| 5 |
+
from .audio_encoder import AudioEncoder
|
| 6 |
+
from .embedder import MultimodalEmbedder
|
| 7 |
+
from .image_encoder import ImageEncoder
|
| 8 |
+
from .text_encoder import TextEncoder
|
| 9 |
+
|
| 10 |
+
|
| 11 |
+
class SingleModalityEmbedder(nn.Module):
|
| 12 |
+
"""Full-depth encoders in a shared space, without optional fusion/exit heads."""
|
| 13 |
+
|
| 14 |
+
def __init__(self, component_dir, legacy=False, max_text_length=512, text_pooling="cls"):
|
| 15 |
+
super().__init__()
|
| 16 |
+
if legacy:
|
| 17 |
+
raise ValueError("Inference profile requires native Nano components")
|
| 18 |
+
root = Path(component_dir)
|
| 19 |
+
self.output_dim = 384
|
| 20 |
+
self.normalize = True
|
| 21 |
+
self.max_text_length = max_text_length
|
| 22 |
+
self.text_encoder = TextEncoder(
|
| 23 |
+
root / "text", output_dim=384, normalize=False,
|
| 24 |
+
max_length=max_text_length, enable_layer_outputs=False,
|
| 25 |
+
pooling_mode=text_pooling,
|
| 26 |
+
)
|
| 27 |
+
self.text_encoder.encoder.pooler = None
|
| 28 |
+
self.image_encoder = ImageEncoder(
|
| 29 |
+
root / "image", output_dim=384, normalize=False,
|
| 30 |
+
enable_layer_outputs=False, legacy=False,
|
| 31 |
+
)
|
| 32 |
+
self.audio_encoder = AudioEncoder(
|
| 33 |
+
root / "audio", output_dim=384, normalize=False,
|
| 34 |
+
enable_layer_outputs=False,
|
| 35 |
+
)
|
| 36 |
+
|
| 37 |
+
def encode_text(self, texts):
|
| 38 |
+
return MultimodalEmbedder.encode_text(self, texts)
|
| 39 |
+
|
| 40 |
+
def encode_image(self, images):
|
| 41 |
+
return MultimodalEmbedder.encode_image(self, images)
|
| 42 |
+
|
| 43 |
+
def encode_audio(self, audio, sampling_rate=16000):
|
| 44 |
+
return MultimodalEmbedder.encode_audio(self, audio, sampling_rate=sampling_rate)
|
| 45 |
+
|
scores.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
| 1 |
{
|
| 2 |
"schema_version": 2,
|
| 3 |
"model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 4 |
-
"revision_semantics": "self denotes
|
| 5 |
"score_scale": "0-1; multiply by 100 for percentages",
|
| 6 |
"matrix": {
|
| 7 |
"small": {
|
|
@@ -84,7 +84,7 @@
|
|
| 84 |
"candidate": {
|
| 85 |
"model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 86 |
"revision": "self",
|
| 87 |
-
"artifact_fingerprint_sha256": "
|
| 88 |
"metrics": {
|
| 89 |
"audio.audio_to_text.recall@1": {
|
| 90 |
"denominator": 2611,
|
|
@@ -157,7 +157,11 @@
|
|
| 157 |
"value": 0.819650067294751
|
| 158 |
}
|
| 159 |
},
|
| 160 |
-
"parameters":
|
|
|
|
|
|
|
|
|
|
|
|
|
| 161 |
}
|
| 162 |
},
|
| 163 |
"datasets": {
|
|
@@ -475,10 +479,15 @@
|
|
| 475 |
"nano": {
|
| 476 |
"repo_id": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 477 |
"revision": "self",
|
| 478 |
-
"native_artifact_sha256": "
|
| 479 |
-
"parameters":
|
| 480 |
"embedding_dimensions": 384,
|
| 481 |
-
"max_text_tokens": 512
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 482 |
},
|
| 483 |
"mini": {
|
| 484 |
"repo_id": "llm-semantic-router/Vela-1.0-Omni-Mini",
|
|
@@ -2624,14 +2633,14 @@
|
|
| 2624 |
"tokens_include_special_tokens": true,
|
| 2625 |
"public_text_overflow": "Raises ValueError; shorten or explicitly chunk longer inputs.",
|
| 2626 |
"embedding_dimensions": 384,
|
| 2627 |
-
"parameters":
|
| 2628 |
},
|
| 2629 |
"efficiency": {
|
| 2630 |
"status": "not_measured_for_this_release",
|
| 2631 |
-
"parameters":
|
| 2632 |
"embedding_dimensions": 384,
|
| 2633 |
"dense_fp32_index_bytes_per_vector": 1536,
|
| 2634 |
-
"weights_safetensors_bytes":
|
| 2635 |
"scope": "No new latency or peak-memory measurement is claimed. Vector storage excludes index structures and metadata.",
|
| 2636 |
"historical_measurement": {
|
| 2637 |
"attention": "native SDPA",
|
|
@@ -2669,6 +2678,14 @@
|
|
| 2669 |
"warmup_batches": 2,
|
| 2670 |
"status": "historical_previous_release_measurement",
|
| 2671 |
"applicability": "Historical measurement on its recorded native artifact; not applicable to this text-backbone update."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2672 |
}
|
| 2673 |
},
|
| 2674 |
"previous_release": {
|
|
@@ -3725,6 +3742,13 @@
|
|
| 3725 |
"report": "benchmarks/component-equivalence.json",
|
| 3726 |
"new_full_panel_inference": false,
|
| 3727 |
"basis": "199 exact text tensors and byte-identical inference/tokenizer/configuration under the same official evaluator protocol."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3728 |
}
|
| 3729 |
},
|
| 3730 |
{
|
|
@@ -3754,7 +3778,14 @@
|
|
| 3754 |
"definition": "Official TaskResult aggregation within each task, followed by official mean_task, mean_task_type and task_types aggregation. The 134 subset/split slots are not averaged as independent tasks.",
|
| 3755 |
"score_scale": "0-1; multiply by 100 for display"
|
| 3756 |
},
|
| 3757 |
-
"report": "benchmarks/maeb-audio-only.json"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3758 |
}
|
| 3759 |
],
|
| 3760 |
"net_benefit": {
|
|
@@ -4295,7 +4326,21 @@
|
|
| 4295 |
"VoxPopuliGenderClustering": 0.0,
|
| 4296 |
"VoxPopuliLanguageID": 0.0
|
| 4297 |
},
|
| 4298 |
-
"paired_interval_scope": "Paired intervals cover the original common metrics, family and two-primary-per-family utility. They are not confidence intervals for the new all-14 common mean or the four-component product aggregate."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4299 |
},
|
| 4300 |
-
"paired_uncertainty_scope": "Paired intervals cover the original common metrics, family and two-primary-per-family utility. They are not confidence intervals for the new all-14 common mean or the four-component product aggregate."
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 4301 |
}
|
|
|
|
| 1 |
{
|
| 2 |
"schema_version": 2,
|
| 3 |
"model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 4 |
+
"revision_semantics": "self denotes this published inference package; evaluated_* identities retain original measurements. The exact public-inference equivalence report binds score applicability.",
|
| 5 |
"score_scale": "0-1; multiply by 100 for percentages",
|
| 6 |
"matrix": {
|
| 7 |
"small": {
|
|
|
|
| 84 |
"candidate": {
|
| 85 |
"model": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 86 |
"revision": "self",
|
| 87 |
+
"artifact_fingerprint_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 88 |
"metrics": {
|
| 89 |
"audio.audio_to_text.recall@1": {
|
| 90 |
"denominator": 2611,
|
|
|
|
| 157 |
"value": 0.819650067294751
|
| 158 |
}
|
| 159 |
},
|
| 160 |
+
"parameters": 135383808,
|
| 161 |
+
"evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 162 |
+
"evaluated_revision": "de02c367bc9aa7a6f7ad3e59fb38962502e021c1",
|
| 163 |
+
"evaluated_parameters": 144554112,
|
| 164 |
+
"inference_equivalence_report": "benchmarks/inference-equivalence.json"
|
| 165 |
}
|
| 166 |
},
|
| 167 |
"datasets": {
|
|
|
|
| 479 |
"nano": {
|
| 480 |
"repo_id": "llm-semantic-router/Vela-1.0-Omni-Nano",
|
| 481 |
"revision": "self",
|
| 482 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 483 |
+
"parameters": 135383808,
|
| 484 |
"embedding_dimensions": 384,
|
| 485 |
+
"max_text_tokens": 512,
|
| 486 |
+
"evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
|
| 487 |
+
"evaluated_revision": "de02c367bc9aa7a6f7ad3e59fb38962502e021c1",
|
| 488 |
+
"evaluated_parameters": 144554112,
|
| 489 |
+
"inference_equivalence_report": "benchmarks/inference-equivalence.json",
|
| 490 |
+
"revision_kind": "HF_snapshot"
|
| 491 |
},
|
| 492 |
"mini": {
|
| 493 |
"repo_id": "llm-semantic-router/Vela-1.0-Omni-Mini",
|
|
|
|
| 2633 |
"tokens_include_special_tokens": true,
|
| 2634 |
"public_text_overflow": "Raises ValueError; shorten or explicitly chunk longer inputs.",
|
| 2635 |
"embedding_dimensions": 384,
|
| 2636 |
+
"parameters": 135383808
|
| 2637 |
},
|
| 2638 |
"efficiency": {
|
| 2639 |
"status": "not_measured_for_this_release",
|
| 2640 |
+
"parameters": 135383808,
|
| 2641 |
"embedding_dimensions": 384,
|
| 2642 |
"dense_fp32_index_bytes_per_vector": 1536,
|
| 2643 |
+
"weights_safetensors_bytes": 541598024,
|
| 2644 |
"scope": "No new latency or peak-memory measurement is claimed. Vector storage excludes index structures and metadata.",
|
| 2645 |
"historical_measurement": {
|
| 2646 |
"attention": "native SDPA",
|
|
|
|
| 2678 |
"warmup_batches": 2,
|
| 2679 |
"status": "historical_previous_release_measurement",
|
| 2680 |
"applicability": "Historical measurement on its recorded native artifact; not applicable to this text-backbone update."
|
| 2681 |
+
},
|
| 2682 |
+
"parameter_storage_change": {
|
| 2683 |
+
"old_safetensors_bytes": 578289664,
|
| 2684 |
+
"new_safetensors_bytes": 541598024,
|
| 2685 |
+
"reduction_bytes": 36691640,
|
| 2686 |
+
"old_FP32_parameter_bytes": 578216448,
|
| 2687 |
+
"new_FP32_parameter_bytes": 541535232,
|
| 2688 |
+
"scope": "Parameter storage only; not measured process peak memory or latency. Vector and index dimensions are unchanged."
|
| 2689 |
}
|
| 2690 |
},
|
| 2691 |
"previous_release": {
|
|
|
|
| 3742 |
"report": "benchmarks/component-equivalence.json",
|
| 3743 |
"new_full_panel_inference": false,
|
| 3744 |
"basis": "199 exact text tensors and byte-identical inference/tokenizer/configuration under the same official evaluator protocol."
|
| 3745 |
+
},
|
| 3746 |
+
"inference_profile_applicability": {
|
| 3747 |
+
"revision": "self",
|
| 3748 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 3749 |
+
"parameters": 135383808,
|
| 3750 |
+
"report": "benchmarks/inference-equivalence.json",
|
| 3751 |
+
"new_benchmark_run": false
|
| 3752 |
}
|
| 3753 |
},
|
| 3754 |
{
|
|
|
|
| 3778 |
"definition": "Official TaskResult aggregation within each task, followed by official mean_task, mean_task_type and task_types aggregation. The 134 subset/split slots are not averaged as independent tasks.",
|
| 3779 |
"score_scale": "0-1; multiply by 100 for display"
|
| 3780 |
},
|
| 3781 |
+
"report": "benchmarks/maeb-audio-only.json",
|
| 3782 |
+
"inference_profile_applicability": {
|
| 3783 |
+
"revision": "self",
|
| 3784 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 3785 |
+
"parameters": 135383808,
|
| 3786 |
+
"report": "benchmarks/inference-equivalence.json",
|
| 3787 |
+
"new_benchmark_run": false
|
| 3788 |
+
}
|
| 3789 |
}
|
| 3790 |
],
|
| 3791 |
"net_benefit": {
|
|
|
|
| 4326 |
"VoxPopuliGenderClustering": 0.0,
|
| 4327 |
"VoxPopuliLanguageID": 0.0
|
| 4328 |
},
|
| 4329 |
+
"paired_interval_scope": "Paired intervals cover the original common metrics, family and two-primary-per-family utility. They are not confidence intervals for the new all-14 common mean or the four-component product aggregate.",
|
| 4330 |
+
"inference_profile_applicability": {
|
| 4331 |
+
"revision": "self",
|
| 4332 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 4333 |
+
"parameters": 135383808,
|
| 4334 |
+
"report": "benchmarks/inference-equivalence.json",
|
| 4335 |
+
"new_benchmark_run": false
|
| 4336 |
+
}
|
| 4337 |
},
|
| 4338 |
+
"paired_uncertainty_scope": "Paired intervals cover the original common metrics, family and two-primary-per-family utility. They are not confidence intervals for the new all-14 common mean or the four-component product aggregate.",
|
| 4339 |
+
"inference_profile": {
|
| 4340 |
+
"revision": "self",
|
| 4341 |
+
"native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
|
| 4342 |
+
"parameters": 135383808,
|
| 4343 |
+
"report": "benchmarks/inference-equivalence.json",
|
| 4344 |
+
"new_benchmark_run": false
|
| 4345 |
+
}
|
| 4346 |
}
|
vela_omni.py
CHANGED
|
@@ -10,13 +10,17 @@ from torch.nn import functional as F
|
|
| 10 |
|
| 11 |
from omni_components.residual_projection import ResidualTextProjection, duration_mean
|
| 12 |
from omni_components.embedder import MultimodalEmbedder
|
|
|
|
| 13 |
from omni_components.mini import MultiModalSentenceEmbedder
|
| 14 |
|
| 15 |
|
| 16 |
class VelaOmni(nn.Module):
|
| 17 |
-
def __init__(self, root, variant, legacy=False, audio_projection="identity", text_projection="identity", audio_pooling="mean", max_text_length=None, text_pooling="mean"):
|
| 18 |
super().__init__()
|
| 19 |
self.variant = variant
|
|
|
|
|
|
|
|
|
|
| 20 |
if text_projection not in ("identity", "residual_rank64"):
|
| 21 |
raise ValueError("Unsupported text projection")
|
| 22 |
if audio_pooling not in ("mean", "duration_mean") or (variant != "nano" and audio_pooling != "mean"):
|
|
@@ -35,7 +39,8 @@ class VelaOmni(nn.Module):
|
|
| 35 |
raise ValueError(f"Unsupported {variant} max_text_length: {max_text_length}")
|
| 36 |
self.max_text_length = max_text_length
|
| 37 |
if variant == "nano":
|
| 38 |
-
|
|
|
|
| 39 |
max_text_length=self.max_text_length, text_pooling=self.text_pooling)
|
| 40 |
self.tokenizer = self.model.text_encoder.tokenizer
|
| 41 |
if text_projection == "residual_rank64":
|
|
@@ -62,15 +67,20 @@ class VelaOmni(nn.Module):
|
|
| 62 |
root = Path(snapshot_download(repo_id, revision=revision,
|
| 63 |
allow_patterns=["config.json", "model.safetensors", "components/*"]))
|
| 64 |
config = json.loads((root / "config.json").read_text())
|
| 65 |
-
if config.get("format_version") not in (1, 2, 3, 4):
|
| 66 |
raise ValueError("Unsupported Vela Omni artifact format")
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
if config.get("image_pooling", "mean") != "mean":
|
| 68 |
raise ValueError("Unsupported image pooling")
|
| 69 |
instance = cls(root, config["variant"], audio_projection=config.get("audio_projection", "identity"),
|
| 70 |
text_projection=config.get("text_projection", "identity"),
|
| 71 |
audio_pooling=config.get("audio_pooling", "mean"),
|
| 72 |
max_text_length=config.get("max_text_length"),
|
| 73 |
-
text_pooling=config.get("text_pooling", "mean")
|
|
|
|
| 74 |
weights = load_file(str(root / "model.safetensors"), device="cpu")
|
| 75 |
result = instance.model.load_state_dict(weights, strict=True, assign=True)
|
| 76 |
if result.missing_keys or result.unexpected_keys:
|
|
|
|
| 10 |
|
| 11 |
from omni_components.residual_projection import ResidualTextProjection, duration_mean
|
| 12 |
from omni_components.embedder import MultimodalEmbedder
|
| 13 |
+
from omni_components.single_modality import SingleModalityEmbedder
|
| 14 |
from omni_components.mini import MultiModalSentenceEmbedder
|
| 15 |
|
| 16 |
|
| 17 |
class VelaOmni(nn.Module):
|
| 18 |
+
def __init__(self, root, variant, legacy=False, audio_projection="identity", text_projection="identity", audio_pooling="mean", max_text_length=None, text_pooling="mean", inference_profile="full"):
|
| 19 |
super().__init__()
|
| 20 |
self.variant = variant
|
| 21 |
+
if inference_profile not in ("full", "single_modality") or (inference_profile == "single_modality" and variant != "nano"):
|
| 22 |
+
raise ValueError("Unsupported inference profile")
|
| 23 |
+
self.inference_profile = inference_profile
|
| 24 |
if text_projection not in ("identity", "residual_rank64"):
|
| 25 |
raise ValueError("Unsupported text projection")
|
| 26 |
if audio_pooling not in ("mean", "duration_mean") or (variant != "nano" and audio_pooling != "mean"):
|
|
|
|
| 39 |
raise ValueError(f"Unsupported {variant} max_text_length: {max_text_length}")
|
| 40 |
self.max_text_length = max_text_length
|
| 41 |
if variant == "nano":
|
| 42 |
+
encoder_type = SingleModalityEmbedder if inference_profile == "single_modality" else MultimodalEmbedder
|
| 43 |
+
self.model = encoder_type(Path(root) / "components", legacy=legacy,
|
| 44 |
max_text_length=self.max_text_length, text_pooling=self.text_pooling)
|
| 45 |
self.tokenizer = self.model.text_encoder.tokenizer
|
| 46 |
if text_projection == "residual_rank64":
|
|
|
|
| 67 |
root = Path(snapshot_download(repo_id, revision=revision,
|
| 68 |
allow_patterns=["config.json", "model.safetensors", "components/*"]))
|
| 69 |
config = json.loads((root / "config.json").read_text())
|
| 70 |
+
if config.get("format_version") not in (1, 2, 3, 4, 5):
|
| 71 |
raise ValueError("Unsupported Vela Omni artifact format")
|
| 72 |
+
if config.get("format_version") == 5 and (config.get("variant") != "nano" or config.get("inference_profile") != "single_modality"):
|
| 73 |
+
raise ValueError("Format 5 requires Nano single-modality profile")
|
| 74 |
+
if config.get("format_version") != 5 and config.get("inference_profile", "full") != "full":
|
| 75 |
+
raise ValueError("Inference profile requires format 5")
|
| 76 |
if config.get("image_pooling", "mean") != "mean":
|
| 77 |
raise ValueError("Unsupported image pooling")
|
| 78 |
instance = cls(root, config["variant"], audio_projection=config.get("audio_projection", "identity"),
|
| 79 |
text_projection=config.get("text_projection", "identity"),
|
| 80 |
audio_pooling=config.get("audio_pooling", "mean"),
|
| 81 |
max_text_length=config.get("max_text_length"),
|
| 82 |
+
text_pooling=config.get("text_pooling", "mean"),
|
| 83 |
+
inference_profile=config.get("inference_profile", "full"))
|
| 84 |
weights = load_file(str(root / "model.safetensors"), device="cpu")
|
| 85 |
result = instance.model.load_state_dict(weights, strict=True, assign=True)
|
| 86 |
if result.missing_keys or result.unexpected_keys:
|