Xunzhuo commited on
Commit
81d2b82
·
verified ·
1 Parent(s): de02c36

Reduce Omni Nano size with equivalent embeddings

Browse files
README.md CHANGED
@@ -31,18 +31,20 @@ Text, images and speech in one embedding space. Vela Omni Nano supports multimod
31
  | Feature | Value |
32
  | --- | --- |
33
  | Modalities | Text, images, speech |
34
- | Total parameters | 144.6M (144,554,112) |
35
  | Embedding dimensions | 384 |
36
  | Text context | 512 tokens, including special tokens |
37
  | Audio input | Mono, 16 kHz, up to 30 seconds |
38
  | Output | L2-normalized vectors; cosine similarity |
39
  | License | Apache 2.0; see component attribution |
40
 
 
 
41
  ## Evaluation
42
 
43
- MTEB 2.21.0; scores are 0–100. **Bold marks gains over the [previous Nano](https://huggingface.co/llm-semantic-router/Vela-1.0-Omni-Nano/blob/3a1efc48995e2b71622d65c6cf71cfc00bbf3aa3/README.md).** Both full panels retain every task and split.
44
 
45
- | Benchmark / metric | Previous Nano | Current Nano |
46
  | --- | ---: | ---: |
47
  | English v2 · 41/41 · mean task | 53.37 | **64.88** |
48
  | English v2 · mean task type | 51.98 | **60.78** |
@@ -53,7 +55,7 @@ Text and speech–text retrieval improve; image classification and clustering ar
53
 
54
  ### Size and quality
55
 
56
- Nano scores **64.89 ArXiv V-measure**, **63.28 NMSQA max AP** and **13.80 Vehicle V-measure**, at 144.6M total parameters.
57
 
58
  <table>
59
  <tr>
 
31
  | Feature | Value |
32
  | --- | --- |
33
  | Modalities | Text, images, speech |
34
+ | Total parameters | 135.4M (135,383,808) |
35
  | Embedding dimensions | 384 |
36
  | Text context | 512 tokens, including special tokens |
37
  | Audio input | Mono, 16 kHz, up to 30 seconds |
38
  | Output | L2-normalized vectors; cosine similarity |
39
  | License | Apache 2.0; see component attribution |
40
 
41
+ The smaller inference package preserves the encoding computation and all reported scores. [Equivalence and storage details](benchmarks/inference-equivalence.json).
42
+
43
  ## Evaluation
44
 
45
+ MTEB 2.21.0; scores are 0–100. **Bold marks gains over the [reference Nano](https://huggingface.co/llm-semantic-router/Vela-1.0-Omni-Nano/blob/3a1efc48995e2b71622d65c6cf71cfc00bbf3aa3/README.md).** Both full panels retain every task and split.
46
 
47
+ | Benchmark / metric | Reference Nano | Current Nano |
48
  | --- | ---: | ---: |
49
  | English v2 · 41/41 · mean task | 53.37 | **64.88** |
50
  | English v2 · mean task type | 51.98 | **60.78** |
 
55
 
56
  ### Size and quality
57
 
58
+ Nano scores **64.89 ArXiv V-measure**, **63.28 NMSQA max AP** and **13.80 Vehicle V-measure**, at 135.4M total parameters.
59
 
60
  <table>
61
  <tr>
assets/pareto-arxiv.png CHANGED

Git LFS Details

  • SHA256: 2933257e65f17a0e607af2c27831b69a1a87cb5e80f85f7b295600383fadba41
  • Pointer size: 131 Bytes
  • Size of remote file: 292 kB

Git LFS Details

  • SHA256: cbcac9f6f7b76a00dca4f8f1929d488a4f956032a872094da906172c4fffc8d6
  • Pointer size: 131 Bytes
  • Size of remote file: 292 kB
assets/pareto-arxiv.svg CHANGED
assets/pareto-nmsqa.png CHANGED

Git LFS Details

  • SHA256: 70f634531a8c33025e80ceec83df5e738371ca53173d09afae61821802112e36
  • Pointer size: 131 Bytes
  • Size of remote file: 255 kB

Git LFS Details

  • SHA256: 41254bf01858573032b205785f9a6fb114b1e72300128a8f1299059b8b1f3824
  • Pointer size: 131 Bytes
  • Size of remote file: 254 kB
assets/pareto-nmsqa.svg CHANGED
assets/pareto-sibfleurs.png CHANGED

Git LFS Details

  • SHA256: 3ebcf8bc85b959b3e7ab78ba0895b0454a553b080c1685d8e98c1b84e8dcd405
  • Pointer size: 131 Bytes
  • Size of remote file: 246 kB

Git LFS Details

  • SHA256: a6dae74c972711c17f4ab060d1d793599738db83fc4cadab215ece902d54898f
  • Pointer size: 131 Bytes
  • Size of remote file: 246 kB
assets/pareto-sibfleurs.svg CHANGED
assets/pareto-vehiclesoundclustering.png CHANGED

Git LFS Details

  • SHA256: 43c4061687d4e71e6309ab047d5ce788156cad3c2a92b7fbacbcda8e4c55fc23
  • Pointer size: 131 Bytes
  • Size of remote file: 259 kB

Git LFS Details

  • SHA256: 2ccf77e6d8c3eede0338a68679833c0c08983ae6de3bf9c80b58b5475e7a00a2
  • Pointer size: 131 Bytes
  • Size of remote file: 259 kB
assets/pareto-vehiclesoundclustering.svg CHANGED
benchmarks/EVALUATION.md CHANGED
@@ -1,6 +1,6 @@
1
  # Evaluation details
2
 
3
- This report identifies Nano artifact `96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4` (144,554,112 parameters). The previous Nano is [3a1efc48](https://huggingface.co/llm-semantic-router/Vela-1.0-Omni-Nano/tree/3a1efc48995e2b71622d65c6cf71cfc00bbf3aa3) (`fb9cd39c9980a085b75456c715c2295b997784fe733314be68c13635f6384edf`). Scores are shown on a 0–100 scale. Bold means an unrounded improvement over the explicitly named comparator.
4
 
5
  ## Complete panels
6
 
@@ -11,7 +11,7 @@ English contains all 41 tasks and seven task types. Audio contains all 19 audio-
11
  | English v2 · 41/41 | 53.37 | **64.88** | 51.98 | **60.78** |
12
  | MAEB audio-only · 19/19 | 39.78 | 39.77 | 46.97 | 46.97 |
13
 
14
- English was evaluated on an earlier native artifact with exactly the same 199 GIST text tensors and identical source/configuration/tokenizer bytes. Its actual evaluated identity is retained in [the full report](mteb-eng-v2.json) and [text equivalence evidence](component-equivalence.json). The current image, audio and cross-modal scores use fresh current-artifact inference.
15
 
16
  ### All English tasks
17
 
@@ -148,8 +148,12 @@ Regressions remain visible: all three image-to-text recall ranks decline; Pets z
148
 
149
  The GIST foundation card discloses MTEB classification training data. We do not claim absence of pretraining overlap. Cross-modal optimization uses COCO and LibriSpeech TRAIN only with complete positives and no benchmark/Studio-example training. One final 300-step pair of orthogonal media rotations was exported; text and media encoders are frozen.
150
 
151
- The current model has 144,554,112 parameters and 384-dimensional FP32 vectors (1,536 raw bytes per vector, excluding index overhead). No current-artifact latency or peak-memory claim is made. Historical measurements in scores.json retain their measured artifact identity.
152
 
153
  ## Task-level Pareto comparisons
154
 
155
  [Every plotted observation](pareto-data.json) and [methodology](pareto-methodology.md) retain the reported API snapshot and separate independent peer reruns. F2 reaches 69.7422 ArXiv V-measure and AST reaches 14.3429 Vehicle V-measure. Nano is not on either updated frontier. Parameter counts refer to whole stored models, including all Vela modalities. These four tasks do not establish overall size SOTA.
 
 
 
 
 
1
  # Evaluation details
2
 
3
+ This report retains measurements from Nano artifact `96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4` (144,554,112 parameters). The previous Nano is [3a1efc48](https://huggingface.co/llm-semantic-router/Vela-1.0-Omni-Nano/tree/3a1efc48995e2b71622d65c6cf71cfc00bbf3aa3) (`fb9cd39c9980a085b75456c715c2295b997784fe733314be68c13635f6384edf`). Scores are shown on a 0–100 scale. Bold means an unrounded improvement over the explicitly named comparator.
4
 
5
  ## Complete panels
6
 
 
11
  | English v2 · 41/41 | 53.37 | **64.88** | 51.98 | **60.78** |
12
  | MAEB audio-only · 19/19 | 39.78 | 39.77 | 46.97 | 46.97 |
13
 
14
+ English was evaluated on an earlier native artifact with exactly the same 199 GIST text tensors and identical source/configuration/tokenizer bytes. Its actual evaluated identity is retained in [the full report](mteb-eng-v2.json) and [text equivalence evidence](component-equivalence.json). Image, audio and cross-modal scores were measured on the 96d artifact. The 135,383,808-parameter inference package preserves the participating weights and public computation; [the equivalence report](inference-equivalence.json) records the exact identities and CPU/GPU synthetic checks. These are applicability checks, not a new benchmark run.
15
 
16
  ### All English tasks
17
 
 
148
 
149
  The GIST foundation card discloses MTEB classification training data. We do not claim absence of pretraining overlap. Cross-modal optimization uses COCO and LibriSpeech TRAIN only with complete positives and no benchmark/Studio-example training. One final 300-step pair of orthogonal media rotations was exported; text and media encoders are frozen.
150
 
151
+ The evaluated artifact has 144,554,112 parameters; the current inference package has 135,383,808 parameters and 384-dimensional FP32 vectors (1,536 raw bytes per vector, excluding index overhead). No current-artifact latency or peak-memory claim is made. Historical measurements in scores.json retain their measured artifact identity.
152
 
153
  ## Task-level Pareto comparisons
154
 
155
  [Every plotted observation](pareto-data.json) and [methodology](pareto-methodology.md) retain the reported API snapshot and separate independent peer reruns. F2 reaches 69.7422 ArXiv V-measure and AST reaches 14.3429 Vehicle V-measure. Nano is not on either updated frontier. Parameter counts refer to whole stored models, including all Vela modalities. These four tasks do not establish overall size SOTA.
156
+
157
+ ## Inference package update
158
+
159
+ The package removes 95 unused tensors (9,170,304 parameters, 6.3439%) while retaining all 476 tensors used by the public encoding paths unchanged. Safetensors storage falls from 578,289,664 to 541,598,024 bytes. Scores, embeddings dimensions, and context limits are unchanged. No latency or measured peak-memory improvement is claimed. The previous quality comparison above keeps its original comparator and measured identities.
benchmarks/component-equivalence.json CHANGED
@@ -26,5 +26,12 @@
26
  "reports": [
27
  "benchmarks/mteb-eng-v2.json"
28
  ],
29
- "scope": "The full English report keeps the artifact that was actually evaluated. Only text-only scores apply through this bridge. Image, audio and cross-modal results are freshly evaluated on the current artifact."
 
 
 
 
 
 
 
30
  }
 
26
  "reports": [
27
  "benchmarks/mteb-eng-v2.json"
28
  ],
29
+ "scope": "The full English report keeps the artifact that was actually evaluated. Only text-only scores apply through this bridge. Image, audio and cross-modal results are freshly evaluated on the current artifact.",
30
+ "inference_profile_applicability": {
31
+ "revision": "self",
32
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
33
+ "parameters": 135383808,
34
+ "report": "benchmarks/inference-equivalence.json",
35
+ "new_benchmark_run": false
36
+ }
37
  }
benchmarks/inference-equivalence.json ADDED
@@ -0,0 +1,453 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "schema_version": 1,
3
+ "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
4
+ "scope": "Same supported public encode_text, encode_image, and encode_audio computation; no new benchmark run.",
5
+ "evaluated_parent": {
6
+ "revision": "de02c367bc9aa7a6f7ad3e59fb38962502e021c1",
7
+ "native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
8
+ "parameters": 144554112
9
+ },
10
+ "applies_to": {
11
+ "revision": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
12
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
13
+ "parameters": 135383808,
14
+ "revision_kind": "native_artifact_fingerprint"
15
+ },
16
+ "removed_unused_tensors": 95,
17
+ "removed_parameters": 9170304,
18
+ "removed_percent": 6.3438555106616406,
19
+ "retained_tensors": 476,
20
+ "retained_values_shapes_and_dtypes_exact": true,
21
+ "removed_tensor_keys": [
22
+ "audio_encoder.layer_norms.0.bias",
23
+ "audio_encoder.layer_norms.0.weight",
24
+ "audio_encoder.layer_norms.1.bias",
25
+ "audio_encoder.layer_norms.1.weight",
26
+ "audio_encoder.layer_norms.2.bias",
27
+ "audio_encoder.layer_norms.2.weight",
28
+ "audio_encoder.layer_norms.3.bias",
29
+ "audio_encoder.layer_norms.3.weight",
30
+ "audio_encoder.layer_projections.0.bias",
31
+ "audio_encoder.layer_projections.0.weight",
32
+ "audio_encoder.layer_projections.1.bias",
33
+ "audio_encoder.layer_projections.1.weight",
34
+ "audio_encoder.layer_projections.2.bias",
35
+ "audio_encoder.layer_projections.2.weight",
36
+ "audio_encoder.layer_projections.3.bias",
37
+ "audio_encoder.layer_projections.3.weight",
38
+ "fusion.fusion_layers.0.ffn.0.bias",
39
+ "fusion.fusion_layers.0.ffn.0.weight",
40
+ "fusion.fusion_layers.0.ffn.3.bias",
41
+ "fusion.fusion_layers.0.ffn.3.weight",
42
+ "fusion.fusion_layers.0.norm1.bias",
43
+ "fusion.fusion_layers.0.norm1.weight",
44
+ "fusion.fusion_layers.0.norm2.bias",
45
+ "fusion.fusion_layers.0.norm2.weight",
46
+ "fusion.fusion_layers.0.self_attention.in_proj_bias",
47
+ "fusion.fusion_layers.0.self_attention.in_proj_weight",
48
+ "fusion.fusion_layers.0.self_attention.out_proj.bias",
49
+ "fusion.fusion_layers.0.self_attention.out_proj.weight",
50
+ "fusion.fusion_layers.1.ffn.0.bias",
51
+ "fusion.fusion_layers.1.ffn.0.weight",
52
+ "fusion.fusion_layers.1.ffn.3.bias",
53
+ "fusion.fusion_layers.1.ffn.3.weight",
54
+ "fusion.fusion_layers.1.norm1.bias",
55
+ "fusion.fusion_layers.1.norm1.weight",
56
+ "fusion.fusion_layers.1.norm2.bias",
57
+ "fusion.fusion_layers.1.norm2.weight",
58
+ "fusion.fusion_layers.1.self_attention.in_proj_bias",
59
+ "fusion.fusion_layers.1.self_attention.in_proj_weight",
60
+ "fusion.fusion_layers.1.self_attention.out_proj.bias",
61
+ "fusion.fusion_layers.1.self_attention.out_proj.weight",
62
+ "fusion.input_projections.audio.projector.0.bias",
63
+ "fusion.input_projections.audio.projector.0.weight",
64
+ "fusion.input_projections.audio.projector.1.bias",
65
+ "fusion.input_projections.audio.projector.1.weight",
66
+ "fusion.input_projections.audio.projector.4.bias",
67
+ "fusion.input_projections.audio.projector.4.weight",
68
+ "fusion.input_projections.image.projector.0.bias",
69
+ "fusion.input_projections.image.projector.0.weight",
70
+ "fusion.input_projections.image.projector.1.bias",
71
+ "fusion.input_projections.image.projector.1.weight",
72
+ "fusion.input_projections.image.projector.4.bias",
73
+ "fusion.input_projections.image.projector.4.weight",
74
+ "fusion.input_projections.text.projector.0.bias",
75
+ "fusion.input_projections.text.projector.0.weight",
76
+ "fusion.input_projections.text.projector.1.bias",
77
+ "fusion.input_projections.text.projector.1.weight",
78
+ "fusion.input_projections.text.projector.4.bias",
79
+ "fusion.input_projections.text.projector.4.weight",
80
+ "fusion.layer_norms.0.bias",
81
+ "fusion.layer_norms.0.weight",
82
+ "fusion.layer_norms.1.bias",
83
+ "fusion.layer_norms.1.weight",
84
+ "fusion.layer_projections.0.bias",
85
+ "fusion.layer_projections.0.weight",
86
+ "fusion.layer_projections.1.bias",
87
+ "fusion.layer_projections.1.weight",
88
+ "fusion.modality_embeddings.weight",
89
+ "fusion.output_projection.bias",
90
+ "fusion.output_projection.weight",
91
+ "image_encoder.layer_projections.0.bias",
92
+ "image_encoder.layer_projections.0.weight",
93
+ "image_encoder.layer_projections.1.bias",
94
+ "image_encoder.layer_projections.1.weight",
95
+ "image_encoder.layer_projections.10.bias",
96
+ "image_encoder.layer_projections.10.weight",
97
+ "image_encoder.layer_projections.11.bias",
98
+ "image_encoder.layer_projections.11.weight",
99
+ "image_encoder.layer_projections.2.bias",
100
+ "image_encoder.layer_projections.2.weight",
101
+ "image_encoder.layer_projections.3.bias",
102
+ "image_encoder.layer_projections.3.weight",
103
+ "image_encoder.layer_projections.4.bias",
104
+ "image_encoder.layer_projections.4.weight",
105
+ "image_encoder.layer_projections.5.bias",
106
+ "image_encoder.layer_projections.5.weight",
107
+ "image_encoder.layer_projections.6.bias",
108
+ "image_encoder.layer_projections.6.weight",
109
+ "image_encoder.layer_projections.7.bias",
110
+ "image_encoder.layer_projections.7.weight",
111
+ "image_encoder.layer_projections.8.bias",
112
+ "image_encoder.layer_projections.8.weight",
113
+ "image_encoder.layer_projections.9.bias",
114
+ "image_encoder.layer_projections.9.weight",
115
+ "text_encoder.encoder.pooler.dense.bias",
116
+ "text_encoder.encoder.pooler.dense.weight"
117
+ ],
118
+ "supported_public_api": [
119
+ "encode_text",
120
+ "encode_image",
121
+ "encode_audio"
122
+ ],
123
+ "omitted_internal_features": [
124
+ "fusion",
125
+ "intermediate layer exits"
126
+ ],
127
+ "quality_evidence": "Original common14, selected standard11, English41 and audio19 scores retain their measured artifact identities. English41 first applies from its original 299 artifact to the 96d parent through exact text identity; this public-inference bridge then applies those scores to the smaller package.",
128
+ "text_context_tokens": 512,
129
+ "embedding_dimensions": 384,
130
+ "CPU_synthetic_parity": {
131
+ "all_retained_tensors_exact": true,
132
+ "candidate_fingerprint": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
133
+ "device": "cpu",
134
+ "inputs": "deterministic synthetic fixtures, not benchmark quality evaluation",
135
+ "old_artifact_new_loader": {
136
+ "bitwise": true,
137
+ "maxabs": 0.0,
138
+ "relative_L2": 0.0,
139
+ "shape": [
140
+ 2,
141
+ 384
142
+ ]
143
+ },
144
+ "parameters": 135383808,
145
+ "public_boundaries": {
146
+ "129": {
147
+ "comparison": {
148
+ "bitwise": true,
149
+ "maxabs": 0.0,
150
+ "relative_L2": 0.0,
151
+ "shape": [
152
+ 1,
153
+ 384
154
+ ]
155
+ },
156
+ "tokens": 129
157
+ },
158
+ "512": {
159
+ "comparison": {
160
+ "bitwise": true,
161
+ "maxabs": 0.0,
162
+ "relative_L2": 0.0,
163
+ "shape": [
164
+ 1,
165
+ 384
166
+ ]
167
+ },
168
+ "tokens": 512
169
+ },
170
+ "513_rejected": "Text exceeds 512 tokens; shorten or explicitly chunk it",
171
+ "audio_bad_rate_rejected": "Resample audio to 16000 Hz before encoding",
172
+ "audio_empty_rejected": "Audio must contain finite nonempty mono waveforms",
173
+ "audio_over30_rejected": "Audio inputs must be at most 30 seconds",
174
+ "text_empty_list_rejected": "Provide at least one text string"
175
+ },
176
+ "quality_scores_generated": false,
177
+ "same_loaded_object_removed_branches": {
178
+ "audio5": {
179
+ "bitwise": true,
180
+ "maxabs": 0.0,
181
+ "relative_L2": 0.0,
182
+ "shape": [
183
+ 5,
184
+ 384
185
+ ]
186
+ },
187
+ "image4": {
188
+ "bitwise": true,
189
+ "maxabs": 0.0,
190
+ "relative_L2": 0.0,
191
+ "shape": [
192
+ 4,
193
+ 384
194
+ ]
195
+ },
196
+ "text8": {
197
+ "bitwise": true,
198
+ "maxabs": 0.0,
199
+ "relative_L2": 0.0,
200
+ "shape": [
201
+ 8,
202
+ 384
203
+ ]
204
+ },
205
+ "text_single": {
206
+ "bitwise": true,
207
+ "maxabs": 0.0,
208
+ "relative_L2": 0.0,
209
+ "shape": [
210
+ 1,
211
+ 384
212
+ ]
213
+ }
214
+ },
215
+ "separate_loaded_native": {
216
+ "audio5": {
217
+ "bitwise": true,
218
+ "maxabs": 0.0,
219
+ "relative_L2": 0.0,
220
+ "shape": [
221
+ 5,
222
+ 384
223
+ ]
224
+ },
225
+ "image4": {
226
+ "bitwise": true,
227
+ "maxabs": 0.0,
228
+ "relative_L2": 0.0,
229
+ "shape": [
230
+ 4,
231
+ 384
232
+ ]
233
+ },
234
+ "text8": {
235
+ "bitwise": true,
236
+ "maxabs": 0.0,
237
+ "relative_L2": 0.0,
238
+ "shape": [
239
+ 8,
240
+ 384
241
+ ]
242
+ },
243
+ "text_single": {
244
+ "bitwise": true,
245
+ "maxabs": 0.0,
246
+ "relative_L2": 0.0,
247
+ "shape": [
248
+ 1,
249
+ 384
250
+ ]
251
+ }
252
+ },
253
+ "source_sha256": "28cb093c40cc9b042a9e9873fe1819721c93fc243da14c5ec1c5b654e79de95d",
254
+ "status": "passed",
255
+ "supported_public_api": [
256
+ "encode_text",
257
+ "encode_image",
258
+ "encode_audio"
259
+ ],
260
+ "torch": "2.8.0+cpu",
261
+ "training_steps": 0,
262
+ "unsupported_internal_features": [
263
+ "fusion",
264
+ "intermediate layer exits"
265
+ ]
266
+ },
267
+ "GPU_synthetic_parity": {
268
+ "all_retained_tensors_exact": true,
269
+ "candidate_fingerprint": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
270
+ "device": "cuda",
271
+ "inputs": "deterministic synthetic fixtures, not benchmark quality evaluation",
272
+ "old_artifact_new_loader": {
273
+ "bitwise": true,
274
+ "maxabs": 0.0,
275
+ "relative_L2": 0.0,
276
+ "shape": [
277
+ 2,
278
+ 384
279
+ ]
280
+ },
281
+ "parameters": 135383808,
282
+ "public_boundaries": {
283
+ "129": {
284
+ "comparison": {
285
+ "bitwise": true,
286
+ "maxabs": 0.0,
287
+ "relative_L2": 0.0,
288
+ "shape": [
289
+ 1,
290
+ 384
291
+ ]
292
+ },
293
+ "tokens": 129
294
+ },
295
+ "512": {
296
+ "comparison": {
297
+ "bitwise": true,
298
+ "maxabs": 0.0,
299
+ "relative_L2": 0.0,
300
+ "shape": [
301
+ 1,
302
+ 384
303
+ ]
304
+ },
305
+ "tokens": 512
306
+ },
307
+ "513_rejected": "Text exceeds 512 tokens; shorten or explicitly chunk it",
308
+ "audio_bad_rate_rejected": "Resample audio to 16000 Hz before encoding",
309
+ "audio_empty_rejected": "Audio must contain finite nonempty mono waveforms",
310
+ "audio_over30_rejected": "Audio inputs must be at most 30 seconds",
311
+ "text_empty_list_rejected": "Provide at least one text string"
312
+ },
313
+ "quality_scores_generated": false,
314
+ "same_loaded_object_removed_branches": {
315
+ "audio5": {
316
+ "bitwise": true,
317
+ "maxabs": 0.0,
318
+ "relative_L2": 0.0,
319
+ "shape": [
320
+ 5,
321
+ 384
322
+ ]
323
+ },
324
+ "image4": {
325
+ "bitwise": true,
326
+ "maxabs": 0.0,
327
+ "relative_L2": 0.0,
328
+ "shape": [
329
+ 4,
330
+ 384
331
+ ]
332
+ },
333
+ "text8": {
334
+ "bitwise": true,
335
+ "maxabs": 0.0,
336
+ "relative_L2": 0.0,
337
+ "shape": [
338
+ 8,
339
+ 384
340
+ ]
341
+ },
342
+ "text_single": {
343
+ "bitwise": true,
344
+ "maxabs": 0.0,
345
+ "relative_L2": 0.0,
346
+ "shape": [
347
+ 1,
348
+ 384
349
+ ]
350
+ }
351
+ },
352
+ "separate_loaded_native": {
353
+ "audio5": {
354
+ "bitwise": true,
355
+ "maxabs": 0.0,
356
+ "relative_L2": 0.0,
357
+ "shape": [
358
+ 5,
359
+ 384
360
+ ]
361
+ },
362
+ "image4": {
363
+ "bitwise": true,
364
+ "maxabs": 0.0,
365
+ "relative_L2": 0.0,
366
+ "shape": [
367
+ 4,
368
+ 384
369
+ ]
370
+ },
371
+ "text8": {
372
+ "bitwise": true,
373
+ "maxabs": 0.0,
374
+ "relative_L2": 0.0,
375
+ "shape": [
376
+ 8,
377
+ 384
378
+ ]
379
+ },
380
+ "text_single": {
381
+ "bitwise": true,
382
+ "maxabs": 0.0,
383
+ "relative_L2": 0.0,
384
+ "shape": [
385
+ 1,
386
+ 384
387
+ ]
388
+ }
389
+ },
390
+ "source_sha256": "28cb093c40cc9b042a9e9873fe1819721c93fc243da14c5ec1c5b654e79de95d",
391
+ "status": "passed",
392
+ "supported_public_api": [
393
+ "encode_text",
394
+ "encode_image",
395
+ "encode_audio"
396
+ ],
397
+ "torch": "2.12.0+git6bbd260",
398
+ "training_steps": 0,
399
+ "unsupported_internal_features": [
400
+ "fusion",
401
+ "intermediate layer exits"
402
+ ]
403
+ },
404
+ "cost": {
405
+ "old_safetensors_bytes": 578289664,
406
+ "new_safetensors_bytes": 541598024,
407
+ "reduction_bytes": 36691640,
408
+ "old_FP32_parameter_bytes": 578216448,
409
+ "new_FP32_parameter_bytes": 541535232,
410
+ "scope": "Parameter storage only; not measured process peak memory or latency. Vector and index dimensions are unchanged."
411
+ },
412
+ "native_artifact_files_sha256": {
413
+ "LICENSE": "c71d239df91726fc519c6eb72d318ec65820627232b2f796219e87dcf35d0ab4",
414
+ "NOTICE": "ea26e8a96e7088153afab5ab92d629a12c149d8ecb98d6795d9277b059f6a942",
415
+ "components/audio/config.json": "372c430053183035fb9d2d7079482f10053b6ff7d7367525ada9fdeead8adeac",
416
+ "components/audio/preprocessor_config.json": "9b5cd03a36fbb8a627c64d98a5b5b126ead95a77720723944487311f0110b666",
417
+ "components/image/config.json": "c2b09e8c0a60f3405d3e53897358a8a373bc4286234f30fcc9324d065fc13728",
418
+ "components/image/preprocessor_config.json": "5a0e5062d04603dc0e5eff01440f6bebfc66d396f9ca16e0128b31c5730cafc5",
419
+ "components/text/1_Pooling/config.json": "d1caf60c96f5fba2157c0c26b76d80818fad6cf0b8eb5e73ec372ff9818eba5c",
420
+ "components/text/config.json": "419ef27bf56a60adc670c50610f8099c64b23e4a1912f4ee7e321139518569b2",
421
+ "components/text/config_sentence_transformers.json": "940d5f50db195fa6e5e6a4f122c095f77880de259d74b14a65779ed48bdd7c56",
422
+ "components/text/modules.json": "84e40c8e006c9b1d6c122e02cba9b02458120b5fb0c87b746c41e0207cf642cf",
423
+ "components/text/sentence_bert_config.json": "84e39fda68ccbff05bfa723ae9c0e70e23e2ec373b76e0f8c6e71af72a693cbf",
424
+ "components/text/special_tokens_map.json": "5d5b662e421ea9fac075174bb0688ee0d9431699900b90662acd44b2a350503a",
425
+ "components/text/tokenizer.json": "d241a60d5e8f04cc1b2b3e9ef7a4921b27bf526d9f6050ab90f9267a1f9e5c66",
426
+ "components/text/tokenizer_config.json": "0b29c7bfc889e53b36d9dd3e686dd4300f6525110eaa98c76a5dafceb2029f53",
427
+ "components/text/vocab.txt": "07eced375cec144d27c900241f3e339478dec958f92fddbc551f295c992038a3",
428
+ "config.json": "3be38bb9a59ba29b719aa4885844fd9f353606b2118885e4171507ca65b72c4c",
429
+ "licenses/GIST-MODEL-CARD.md": "e2905b6dc66b39a697cd82f3211eea4a7ce45b17cc720eceae7fdd337b601c3c",
430
+ "model.safetensors": "401759e85cec305aca7ff08b31a1a3efa2c410b24b7bb49f9a81de3aabafa52d",
431
+ "omni_components/__init__.py": "543f72d073d77a71e7f5ad857f26d53abbecbbba1ed094336f5bf730af0bd641",
432
+ "omni_components/audio_encoder.py": "2c1e8e347c3e49a438084cc61f486dda9fde00d9f4b4d7caac828aa9b2f77488",
433
+ "omni_components/audio_io.py": "23392eb0784bd92e4b0176004455025faa0d9d47f6216e12da34079c49765546",
434
+ "omni_components/embedder.py": "ad709fd80bc1850f609443777ddf3327765f3ea330659f5c154f90fa8ab5e8fd",
435
+ "omni_components/fusion.py": "a3f6132f190f1533887a77570f4457a0da4eba698574b90888da477919d5c279",
436
+ "omni_components/image_encoder.py": "40e64a4fde0cb1a1416b764a1a8db47105985e3b979bac7b0aa3bc9d789d0f7a",
437
+ "omni_components/mini.py": "c62439716bcf06a0051a8d9ec7ea41f917816133f425a7fa004561279daac4fe",
438
+ "omni_components/records.py": "acb3a3cda9d260e39007b6aafd74efd002de974700e14a5535f66590b8b801d6",
439
+ "omni_components/residual_projection.py": "30028685698c9b4779c44f8ebe28bf80c1689b99594ca4b287db16d133227ad4",
440
+ "omni_components/single_modality.py": "a61edcdcac3dbad5c6b84101655e290fa5d74d7292bd8f61105772d9b2135171",
441
+ "omni_components/text_backbone.py": "c11a92203f3010ef857d2c809257781150a790acf1877ed83e1a0021449268c3",
442
+ "omni_components/text_encoder.py": "578329b6bf656423e0b9543783f30a6455449a7402fd3a1a9f601a8afc9705af",
443
+ "vela_omni.py": "a04bd0e9b163b0f440128a3086632d0e6223312e592f247f483c221210f08048"
444
+ },
445
+ "evidence_sha256": {
446
+ "BUILD.json": "d3701e04ca5a0949d4da29994a15ea69847c8fbf44fb7ce9757e1ccc1d051373",
447
+ "PARITY_CPU.json": "9d75c24759dc47776a1892b7b9599d2f7a242adb527707ec1054008a1e785540",
448
+ "PARITY_CUDA.json": "99f23ead7e05aea2c30e96361e4d617175fe7290563933c0203db45057938729"
449
+ },
450
+ "training_steps": 0,
451
+ "new_quality_scores": false,
452
+ "new_latency_claim": false
453
+ }
benchmarks/maeb-audio-only.json CHANGED
@@ -11852,5 +11852,12 @@
11852
  "VoxPopuliAccentPairClassification": -0.02844431688273641,
11853
  "VoxPopuliGenderClustering": 0.0,
11854
  "VoxPopuliLanguageID": 0.0
 
 
 
 
 
 
 
11855
  }
11856
  }
 
11852
  "VoxPopuliAccentPairClassification": -0.02844431688273641,
11853
  "VoxPopuliGenderClustering": 0.0,
11854
  "VoxPopuliLanguageID": 0.0
11855
+ },
11856
+ "inference_profile_applicability": {
11857
+ "revision": "self",
11858
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
11859
+ "parameters": 135383808,
11860
+ "report": "benchmarks/inference-equivalence.json",
11861
+ "new_benchmark_run": false
11862
  }
11863
  }
benchmarks/mteb-eng-v2.json CHANGED
@@ -5011,5 +5011,12 @@
5011
  "task": "STS22.v2",
5012
  "task_type": "STS"
5013
  }
5014
- ]
 
 
 
 
 
 
 
5015
  }
 
5011
  "task": "STS22.v2",
5012
  "task_type": "STS"
5013
  }
5014
+ ],
5015
+ "inference_profile_applicability": {
5016
+ "revision": "self",
5017
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
5018
+ "parameters": 135383808,
5019
+ "report": "benchmarks/inference-equivalence.json",
5020
+ "new_benchmark_run": false
5021
+ }
5022
  }
benchmarks/net-progress.json CHANGED
@@ -536,5 +536,12 @@
536
  "VoxPopuliGenderClustering": 0.0,
537
  "VoxPopuliLanguageID": 0.0
538
  },
539
- "paired_interval_scope": "Paired intervals cover the original common metrics, family and two-primary-per-family utility. They are not confidence intervals for the new all-14 common mean or the four-component product aggregate."
 
 
 
 
 
 
 
540
  }
 
536
  "VoxPopuliGenderClustering": 0.0,
537
  "VoxPopuliLanguageID": 0.0
538
  },
539
+ "paired_interval_scope": "Paired intervals cover the original common metrics, family and two-primary-per-family utility. They are not confidence intervals for the new all-14 common mean or the four-component product aggregate.",
540
+ "inference_profile_applicability": {
541
+ "revision": "self",
542
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
543
+ "parameters": 135383808,
544
+ "report": "benchmarks/inference-equivalence.json",
545
+ "new_benchmark_run": false
546
+ }
547
  }
benchmarks/pareto-data.json CHANGED
@@ -2569,8 +2569,8 @@
2569
  },
2570
  {
2571
  "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
2572
- "parameters_billion": 0.144554112,
2573
- "parameters_exact": 144554112,
2574
  "score": 64.88847259326147,
2575
  "open_weights": true,
2576
  "modalities": [
@@ -2582,15 +2582,18 @@
2582
  "kind": "nano",
2583
  "evaluated_task_revision": "0bbdb47bcbe3a90093699aefeed338a0f28a7ee8",
2584
  "on_displayed_frontier": false,
2585
- "evidence_type": "exact_text_component_applicability",
2586
  "observation_id": "vela_release_evaluation:llm-semantic-router/Vela-1.0-Omni-Nano",
2587
- "native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
2588
- "revision": "self",
2589
  "evaluated_native_artifact_sha256": "29994142677559ca657c27b657eda0dddbf583bd4cf42a3a31095ab3f99d1646",
2590
  "evaluated_revision": "29994142677559ca657c27b657eda0dddbf583bd4cf42a3a31095ab3f99d1646",
2591
  "evaluated_model": "local/Vela-Omni-Nano-GIST-Rotation",
2592
  "revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
2593
- "report": "benchmarks/mteb-eng-v2.json"
 
 
 
2594
  },
2595
  {
2596
  "model": "ibm-granite/granite-embedding-english-r2",
@@ -9600,8 +9603,8 @@
9600
  },
9601
  {
9602
  "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
9603
- "parameters_billion": 0.144554112,
9604
- "parameters_exact": 144554112,
9605
  "score": 63.28340215485784,
9606
  "open_weights": true,
9607
  "modalities": [
@@ -9613,14 +9616,17 @@
9613
  "kind": "nano",
9614
  "evaluated_task_revision": "d33eea302d8a64c0a2ee094cf39d77c0814fe87a",
9615
  "on_displayed_frontier": true,
9616
- "native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
9617
- "revision": "self",
9618
  "evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
9619
  "evaluated_revision": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
9620
  "evaluated_model": "llm-semantic-router/Vela-1.0-Omni-Nano",
9621
  "revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
9622
- "evidence_type": "native_evaluation",
9623
- "report": "benchmarks/maeb-audio-only.json"
 
 
 
9624
  },
9625
  {
9626
  "model": "laion/clap-htsat-unfused",
@@ -11233,8 +11239,8 @@
11233
  },
11234
  {
11235
  "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
11236
- "parameters_billion": 0.144554112,
11237
- "parameters_exact": 144554112,
11238
  "score": 18.304657831512046,
11239
  "open_weights": true,
11240
  "modalities": [
@@ -11246,14 +11252,17 @@
11246
  "kind": "nano",
11247
  "evaluated_task_revision": "186b61175fbd77059f769b6bf1110d449a2ff311",
11248
  "on_displayed_frontier": false,
11249
- "native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
11250
- "revision": "self",
11251
  "evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
11252
  "evaluated_revision": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
11253
  "evaluated_model": "llm-semantic-router/Vela-1.0-Omni-Nano",
11254
  "revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
11255
- "evidence_type": "native_evaluation",
11256
- "report": "benchmarks/maeb-audio-only.json"
 
 
 
11257
  },
11258
  {
11259
  "model": "laion/clap-htsat-unfused",
@@ -12912,8 +12921,8 @@
12912
  },
12913
  {
12914
  "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
12915
- "parameters_billion": 0.144554112,
12916
- "parameters_exact": 144554112,
12917
  "score": 13.800701178563807,
12918
  "open_weights": true,
12919
  "modalities": [
@@ -12925,14 +12934,17 @@
12925
  "kind": "nano",
12926
  "evaluated_task_revision": "5f905f9544d0c78b607639662c88c2ddd1216384",
12927
  "on_displayed_frontier": false,
12928
- "native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
12929
- "revision": "self",
12930
  "evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
12931
  "evaluated_revision": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
12932
  "evaluated_model": "llm-semantic-router/Vela-1.0-Omni-Nano",
12933
  "revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
12934
- "evidence_type": "native_evaluation",
12935
- "report": "benchmarks/maeb-audio-only.json"
 
 
 
12936
  },
12937
  {
12938
  "model": "laion/clap-htsat-unfused",
 
2569
  },
2570
  {
2571
  "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
2572
+ "parameters_billion": 0.135383808,
2573
+ "parameters_exact": 135383808,
2574
  "score": 64.88847259326147,
2575
  "open_weights": true,
2576
  "modalities": [
 
2582
  "kind": "nano",
2583
  "evaluated_task_revision": "0bbdb47bcbe3a90093699aefeed338a0f28a7ee8",
2584
  "on_displayed_frontier": false,
2585
+ "evidence_type": "exact_public_inference_applicability",
2586
  "observation_id": "vela_release_evaluation:llm-semantic-router/Vela-1.0-Omni-Nano",
2587
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
2588
+ "revision": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
2589
  "evaluated_native_artifact_sha256": "29994142677559ca657c27b657eda0dddbf583bd4cf42a3a31095ab3f99d1646",
2590
  "evaluated_revision": "29994142677559ca657c27b657eda0dddbf583bd4cf42a3a31095ab3f99d1646",
2591
  "evaluated_model": "local/Vela-Omni-Nano-GIST-Rotation",
2592
  "revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
2593
+ "report": "benchmarks/mteb-eng-v2.json",
2594
+ "prior_applicable_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
2595
+ "inference_equivalence_report": "benchmarks/inference-equivalence.json",
2596
+ "revision_kind": "native_artifact_fingerprint"
2597
  },
2598
  {
2599
  "model": "ibm-granite/granite-embedding-english-r2",
 
9603
  },
9604
  {
9605
  "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
9606
+ "parameters_billion": 0.135383808,
9607
+ "parameters_exact": 135383808,
9608
  "score": 63.28340215485784,
9609
  "open_weights": true,
9610
  "modalities": [
 
9616
  "kind": "nano",
9617
  "evaluated_task_revision": "d33eea302d8a64c0a2ee094cf39d77c0814fe87a",
9618
  "on_displayed_frontier": true,
9619
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
9620
+ "revision": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
9621
  "evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
9622
  "evaluated_revision": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
9623
  "evaluated_model": "llm-semantic-router/Vela-1.0-Omni-Nano",
9624
  "revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
9625
+ "evidence_type": "exact_public_inference_applicability",
9626
+ "report": "benchmarks/maeb-audio-only.json",
9627
+ "prior_applicable_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
9628
+ "inference_equivalence_report": "benchmarks/inference-equivalence.json",
9629
+ "revision_kind": "native_artifact_fingerprint"
9630
  },
9631
  {
9632
  "model": "laion/clap-htsat-unfused",
 
11239
  },
11240
  {
11241
  "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
11242
+ "parameters_billion": 0.135383808,
11243
+ "parameters_exact": 135383808,
11244
  "score": 18.304657831512046,
11245
  "open_weights": true,
11246
  "modalities": [
 
11252
  "kind": "nano",
11253
  "evaluated_task_revision": "186b61175fbd77059f769b6bf1110d449a2ff311",
11254
  "on_displayed_frontier": false,
11255
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
11256
+ "revision": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
11257
  "evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
11258
  "evaluated_revision": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
11259
  "evaluated_model": "llm-semantic-router/Vela-1.0-Omni-Nano",
11260
  "revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
11261
+ "evidence_type": "exact_public_inference_applicability",
11262
+ "report": "benchmarks/maeb-audio-only.json",
11263
+ "prior_applicable_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
11264
+ "inference_equivalence_report": "benchmarks/inference-equivalence.json",
11265
+ "revision_kind": "native_artifact_fingerprint"
11266
  },
11267
  {
11268
  "model": "laion/clap-htsat-unfused",
 
12921
  },
12922
  {
12923
  "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
12924
+ "parameters_billion": 0.135383808,
12925
+ "parameters_exact": 135383808,
12926
  "score": 13.800701178563807,
12927
  "open_weights": true,
12928
  "modalities": [
 
12934
  "kind": "nano",
12935
  "evaluated_task_revision": "5f905f9544d0c78b607639662c88c2ddd1216384",
12936
  "on_displayed_frontier": false,
12937
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
12938
+ "revision": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
12939
  "evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
12940
  "evaluated_revision": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
12941
  "evaluated_model": "llm-semantic-router/Vela-1.0-Omni-Nano",
12942
  "revision_semantics": "Evaluated revisions are native artifact fingerprints; applies-to identity is the current whole artifact.",
12943
+ "evidence_type": "exact_public_inference_applicability",
12944
+ "report": "benchmarks/maeb-audio-only.json",
12945
+ "prior_applicable_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
12946
+ "inference_equivalence_report": "benchmarks/inference-equivalence.json",
12947
+ "revision_kind": "native_artifact_fingerprint"
12948
  },
12949
  {
12950
  "model": "laion/clap-htsat-unfused",
benchmarks/pareto-methodology.md CHANGED
@@ -6,6 +6,6 @@ The frontier uses every displayed observation: a point is dominated if another i
6
 
7
  Reported and independently measured observations remain separate, even for the same model. F2's reported size has an explicit checkpoint-based correction in its report; its actual rerun uses the official task instruction, BF16 and an 8192-token limit. AST's old API point has 87M rounded parameters and no revision; it is not silently normalized. The new AST point counts all 86,594,063 parameters of the pinned checkpoint (86,187,264 execute as the encoder) and uses the official AST processor/raw pooler, FP32, batch eight, all 1,705 Vehicle rows and ten default clustering repetitions with seed 20260918. No training or task tuning is involved. The original NumPy/tensor interface failure and the narrow compatible wrapper are retained in its evidence.
8
 
9
- Nano's English coordinate retains the actual evaluated artifact and applies through exact text identity. Audio coordinates are current-artifact evaluations. All raw identities, parameters, metrics and rich results remain in the linked reports. Within-task API protocol differences and unknown training overlap limit causal comparison. No official benchmark-wide or size-SOTA claim follows from these four task-level plots.
10
 
11
  [All plotted points (CSV)](pareto-points.csv) · [F2 rerun](peer-evaluations/f2llm-v2-80m-arxiv.json) · [AST rerun](peer-evaluations/ast-vehicle.json) · [Complete evaluation](EVALUATION.md)
 
6
 
7
  Reported and independently measured observations remain separate, even for the same model. F2's reported size has an explicit checkpoint-based correction in its report; its actual rerun uses the official task instruction, BF16 and an 8192-token limit. AST's old API point has 87M rounded parameters and no revision; it is not silently normalized. The new AST point counts all 86,594,063 parameters of the pinned checkpoint (86,187,264 execute as the encoder) and uses the official AST processor/raw pooler, FP32, batch eight, all 1,705 Vehicle rows and ten default clustering repetitions with seed 20260918. No training or task tuning is involved. The original NumPy/tensor interface failure and the narrow compatible wrapper are retained in its evidence.
8
 
9
+ Nano's English coordinate retains the actual evaluated artifact and applies through exact text identity. Nano audio coordinates retain their 96d evaluated identity. Nano now has 135,383,808 stored parameters after unused-branch removal; CPU/GPU bitwise synthetic parity and unchanged participating tensors support score applicability through [the public-inference bridge](inference-equivalence.json). No benchmark rerun or score improvement is implied. All raw identities, parameters, metrics and rich results remain in the linked reports. Within-task API protocol differences and unknown training overlap limit causal comparison. No official benchmark-wide or size-SOTA claim follows from these four task-level plots.
10
 
11
  [All plotted points (CSV)](pareto-points.csv) · [F2 rerun](peer-evaluations/f2llm-v2-80m-arxiv.json) · [AST rerun](peer-evaluations/ast-vehicle.json) · [Complete evaluation](EVALUATION.md)
benchmarks/pareto-points.csv CHANGED
@@ -81,7 +81,7 @@ ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1,peer,0.137
81
  ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1-ablated,peer,0.137,57.903400000000005,True,False,registry_reported,registry_reported:nomic-ai/nomic-embed-text-v1-ablated,,
82
  ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1-unsupervised,peer,0.137,56.7453,True,False,registry_reported,registry_reported:nomic-ai/nomic-embed-text-v1-unsupervised,,
83
  ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1.5,peer,0.137,60.08051520875409,True,False,registry_reported,registry_reported:nomic-ai/nomic-embed-text-v1.5,,
84
- ArXivHierarchicalClusteringP2P,v_measure,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.144554112,64.88847259326147,True,False,exact_text_component_applicability,vela_release_evaluation:llm-semantic-router/Vela-1.0-Omni-Nano,self,96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4
85
  ArXivHierarchicalClusteringP2P,v_measure,ibm-granite/granite-embedding-english-r2,peer,0.149,59.0566,True,False,registry_reported,registry_reported:ibm-granite/granite-embedding-english-r2,,
86
  ArXivHierarchicalClusteringP2P,v_measure,Tarka-AIR/Tarka-Embedding-150M-V1,peer,0.156,63.397099999999995,True,False,registry_reported,registry_reported:Tarka-AIR/Tarka-Embedding-150M-V1,,
87
  ArXivHierarchicalClusteringP2P,v_measure,codefuse-ai/F2LLM-v2-160M,peer,0.159,62.4399,True,False,registry_reported,registry_reported:codefuse-ai/F2LLM-v2-160M,,
@@ -234,7 +234,7 @@ NMSQAPairClassification,max_ap,microsoft/wavlm-base-plus-sv,peer,0.095,48.4168,T
234
  NMSQAPairClassification,max_ap,microsoft/wavlm-base-sd,peer,0.095,52.9964,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sd,,
235
  NMSQAPairClassification,max_ap,microsoft/wavlm-base-sv,peer,0.095,52.9964,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sv,,
236
  NMSQAPairClassification,max_ap,asapp/sew-d-mid-400k-ft-ls100h,peer,0.139,48.4208,True,False,registry_reported,registry_reported:asapp/sew-d-mid-400k-ft-ls100h,,
237
- NMSQAPairClassification,max_ap,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.144554112,63.28340215485784,True,True,native_evaluation,,self,96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4
238
  NMSQAPairClassification,max_ap,laion/clap-htsat-unfused,peer,0.153,46.9791,True,False,registry_reported,registry_reported:laion/clap-htsat-unfused,,
239
  NMSQAPairClassification,max_ap,laion/clap-htsat-fused,peer,0.154,47.8507,True,False,registry_reported,registry_reported:laion/clap-htsat-fused,,
240
  NMSQAPairClassification,max_ap,microsoft/msclap-2023,peer,0.16,51.0934,True,False,registry_reported,registry_reported:microsoft/msclap-2023,,
@@ -302,7 +302,7 @@ SIBFLEURS,accuracy,microsoft/wavlm-base-plus-sv,peer,0.095,11.857322549019608,Tr
302
  SIBFLEURS,accuracy,microsoft/wavlm-base-sd,peer,0.095,13.31318431372549,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sd,,
303
  SIBFLEURS,accuracy,microsoft/wavlm-base-sv,peer,0.095,13.31318431372549,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sv,,
304
  SIBFLEURS,accuracy,asapp/sew-d-mid-400k-ft-ls100h,peer,0.139,12.634273529411763,True,False,registry_reported,registry_reported:asapp/sew-d-mid-400k-ft-ls100h,,
305
- SIBFLEURS,accuracy,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.144554112,18.304657831512046,True,False,native_evaluation,,self,96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4
306
  SIBFLEURS,accuracy,laion/clap-htsat-unfused,peer,0.153,8.674730392156862,True,False,registry_reported,registry_reported:laion/clap-htsat-unfused,,
307
  SIBFLEURS,accuracy,laion/clap-htsat-fused,peer,0.154,9.069597058823529,True,False,registry_reported,registry_reported:laion/clap-htsat-fused,,
308
  SIBFLEURS,accuracy,microsoft/msclap-2023,peer,0.16,14.073474509803921,True,False,registry_reported,registry_reported:microsoft/msclap-2023,,
@@ -372,7 +372,7 @@ VehicleSoundClustering,v_measure,microsoft/wavlm-base-plus-sv,peer,0.095,1.7932,
372
  VehicleSoundClustering,v_measure,microsoft/wavlm-base-sd,peer,0.095,0.40940000000000004,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sd,,
373
  VehicleSoundClustering,v_measure,microsoft/wavlm-base-sv,peer,0.095,0.40940000000000004,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sv,,
374
  VehicleSoundClustering,v_measure,asapp/sew-d-mid-400k-ft-ls100h,peer,0.139,0.6276,True,False,registry_reported,registry_reported:asapp/sew-d-mid-400k-ft-ls100h,,
375
- VehicleSoundClustering,v_measure,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.144554112,13.800701178563807,True,False,native_evaluation,,self,96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4
376
  VehicleSoundClustering,v_measure,laion/clap-htsat-unfused,peer,0.153,2.6642,True,False,registry_reported,registry_reported:laion/clap-htsat-unfused,,
377
  VehicleSoundClustering,v_measure,laion/clap-htsat-fused,peer,0.154,4.7132,True,False,registry_reported,registry_reported:laion/clap-htsat-fused,,
378
  VehicleSoundClustering,v_measure,microsoft/msclap-2023,peer,0.16,2.86,True,False,registry_reported,registry_reported:microsoft/msclap-2023,,
 
81
  ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1-ablated,peer,0.137,57.903400000000005,True,False,registry_reported,registry_reported:nomic-ai/nomic-embed-text-v1-ablated,,
82
  ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1-unsupervised,peer,0.137,56.7453,True,False,registry_reported,registry_reported:nomic-ai/nomic-embed-text-v1-unsupervised,,
83
  ArXivHierarchicalClusteringP2P,v_measure,nomic-ai/nomic-embed-text-v1.5,peer,0.137,60.08051520875409,True,False,registry_reported,registry_reported:nomic-ai/nomic-embed-text-v1.5,,
84
+ ArXivHierarchicalClusteringP2P,v_measure,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.135383808,64.88847259326147,True,False,exact_public_inference_applicability,vela_release_evaluation:llm-semantic-router/Vela-1.0-Omni-Nano,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa
85
  ArXivHierarchicalClusteringP2P,v_measure,ibm-granite/granite-embedding-english-r2,peer,0.149,59.0566,True,False,registry_reported,registry_reported:ibm-granite/granite-embedding-english-r2,,
86
  ArXivHierarchicalClusteringP2P,v_measure,Tarka-AIR/Tarka-Embedding-150M-V1,peer,0.156,63.397099999999995,True,False,registry_reported,registry_reported:Tarka-AIR/Tarka-Embedding-150M-V1,,
87
  ArXivHierarchicalClusteringP2P,v_measure,codefuse-ai/F2LLM-v2-160M,peer,0.159,62.4399,True,False,registry_reported,registry_reported:codefuse-ai/F2LLM-v2-160M,,
 
234
  NMSQAPairClassification,max_ap,microsoft/wavlm-base-sd,peer,0.095,52.9964,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sd,,
235
  NMSQAPairClassification,max_ap,microsoft/wavlm-base-sv,peer,0.095,52.9964,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sv,,
236
  NMSQAPairClassification,max_ap,asapp/sew-d-mid-400k-ft-ls100h,peer,0.139,48.4208,True,False,registry_reported,registry_reported:asapp/sew-d-mid-400k-ft-ls100h,,
237
+ NMSQAPairClassification,max_ap,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.135383808,63.28340215485784,True,True,exact_public_inference_applicability,,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa
238
  NMSQAPairClassification,max_ap,laion/clap-htsat-unfused,peer,0.153,46.9791,True,False,registry_reported,registry_reported:laion/clap-htsat-unfused,,
239
  NMSQAPairClassification,max_ap,laion/clap-htsat-fused,peer,0.154,47.8507,True,False,registry_reported,registry_reported:laion/clap-htsat-fused,,
240
  NMSQAPairClassification,max_ap,microsoft/msclap-2023,peer,0.16,51.0934,True,False,registry_reported,registry_reported:microsoft/msclap-2023,,
 
302
  SIBFLEURS,accuracy,microsoft/wavlm-base-sd,peer,0.095,13.31318431372549,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sd,,
303
  SIBFLEURS,accuracy,microsoft/wavlm-base-sv,peer,0.095,13.31318431372549,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sv,,
304
  SIBFLEURS,accuracy,asapp/sew-d-mid-400k-ft-ls100h,peer,0.139,12.634273529411763,True,False,registry_reported,registry_reported:asapp/sew-d-mid-400k-ft-ls100h,,
305
+ SIBFLEURS,accuracy,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.135383808,18.304657831512046,True,False,exact_public_inference_applicability,,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa
306
  SIBFLEURS,accuracy,laion/clap-htsat-unfused,peer,0.153,8.674730392156862,True,False,registry_reported,registry_reported:laion/clap-htsat-unfused,,
307
  SIBFLEURS,accuracy,laion/clap-htsat-fused,peer,0.154,9.069597058823529,True,False,registry_reported,registry_reported:laion/clap-htsat-fused,,
308
  SIBFLEURS,accuracy,microsoft/msclap-2023,peer,0.16,14.073474509803921,True,False,registry_reported,registry_reported:microsoft/msclap-2023,,
 
372
  VehicleSoundClustering,v_measure,microsoft/wavlm-base-sd,peer,0.095,0.40940000000000004,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sd,,
373
  VehicleSoundClustering,v_measure,microsoft/wavlm-base-sv,peer,0.095,0.40940000000000004,True,False,registry_reported,registry_reported:microsoft/wavlm-base-sv,,
374
  VehicleSoundClustering,v_measure,asapp/sew-d-mid-400k-ft-ls100h,peer,0.139,0.6276,True,False,registry_reported,registry_reported:asapp/sew-d-mid-400k-ft-ls100h,,
375
+ VehicleSoundClustering,v_measure,llm-semantic-router/Vela-1.0-Omni-Nano,nano,0.135383808,13.800701178563807,True,False,exact_public_inference_applicability,,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa,50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa
376
  VehicleSoundClustering,v_measure,laion/clap-htsat-unfused,peer,0.153,2.6642,True,False,registry_reported,registry_reported:laion/clap-htsat-unfused,,
377
  VehicleSoundClustering,v_measure,laion/clap-htsat-fused,peer,0.154,4.7132,True,False,registry_reported,registry_reported:laion/clap-htsat-fused,,
378
  VehicleSoundClustering,v_measure,microsoft/msclap-2023,peer,0.16,2.86,True,False,registry_reported,registry_reported:microsoft/msclap-2023,,
config.json CHANGED
@@ -1,18 +1,19 @@
1
  {
2
- "format_version": 4,
3
- "variant": "nano",
4
- "parameter_count": 144554112,
5
- "embedding_dim": 384,
6
- "max_text_length": 512,
7
  "architectures": [
8
  "VelaOmni"
9
  ],
10
- "torch_dtype": "float32",
11
- "library_name": "pytorch",
12
- "audio_sampling_rate": 16000,
13
  "audio_max_seconds": 30,
 
14
  "audio_projection": "linear",
 
 
 
 
 
 
 
 
15
  "text_projection": "identity",
16
- "audio_pooling": "mean",
17
- "text_pooling": "cls"
18
  }
 
1
  {
 
 
 
 
 
2
  "architectures": [
3
  "VelaOmni"
4
  ],
 
 
 
5
  "audio_max_seconds": 30,
6
+ "audio_pooling": "mean",
7
  "audio_projection": "linear",
8
+ "audio_sampling_rate": 16000,
9
+ "embedding_dim": 384,
10
+ "format_version": 5,
11
+ "inference_profile": "single_modality",
12
+ "library_name": "pytorch",
13
+ "max_text_length": 512,
14
+ "parameter_count": 135383808,
15
+ "text_pooling": "cls",
16
  "text_projection": "identity",
17
+ "torch_dtype": "float32",
18
+ "variant": "nano"
19
  }
model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:0d6fc952cbf4d99d7e5dcb6573ea726372aa460d2dde076141e1682728e7cd6f
3
- size 578289664
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:401759e85cec305aca7ff08b31a1a3efa2c410b24b7bb49f9a81de3aabafa52d
3
+ size 541598024
omni_components/single_modality.py ADDED
@@ -0,0 +1,45 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ """Nano inference runtime for the three public modality encoders."""
2
+ from pathlib import Path
3
+ from torch import nn
4
+
5
+ from .audio_encoder import AudioEncoder
6
+ from .embedder import MultimodalEmbedder
7
+ from .image_encoder import ImageEncoder
8
+ from .text_encoder import TextEncoder
9
+
10
+
11
+ class SingleModalityEmbedder(nn.Module):
12
+ """Full-depth encoders in a shared space, without optional fusion/exit heads."""
13
+
14
+ def __init__(self, component_dir, legacy=False, max_text_length=512, text_pooling="cls"):
15
+ super().__init__()
16
+ if legacy:
17
+ raise ValueError("Inference profile requires native Nano components")
18
+ root = Path(component_dir)
19
+ self.output_dim = 384
20
+ self.normalize = True
21
+ self.max_text_length = max_text_length
22
+ self.text_encoder = TextEncoder(
23
+ root / "text", output_dim=384, normalize=False,
24
+ max_length=max_text_length, enable_layer_outputs=False,
25
+ pooling_mode=text_pooling,
26
+ )
27
+ self.text_encoder.encoder.pooler = None
28
+ self.image_encoder = ImageEncoder(
29
+ root / "image", output_dim=384, normalize=False,
30
+ enable_layer_outputs=False, legacy=False,
31
+ )
32
+ self.audio_encoder = AudioEncoder(
33
+ root / "audio", output_dim=384, normalize=False,
34
+ enable_layer_outputs=False,
35
+ )
36
+
37
+ def encode_text(self, texts):
38
+ return MultimodalEmbedder.encode_text(self, texts)
39
+
40
+ def encode_image(self, images):
41
+ return MultimodalEmbedder.encode_image(self, images)
42
+
43
+ def encode_audio(self, audio, sampling_rate=16000):
44
+ return MultimodalEmbedder.encode_audio(self, audio, sampling_rate=sampling_rate)
45
+
scores.json CHANGED
@@ -1,7 +1,7 @@
1
  {
2
  "schema_version": 2,
3
  "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
4
- "revision_semantics": "self denotes the Hugging Face snapshot containing this file; the native artifact fingerprint binds its evaluated model files.",
5
  "score_scale": "0-1; multiply by 100 for percentages",
6
  "matrix": {
7
  "small": {
@@ -84,7 +84,7 @@
84
  "candidate": {
85
  "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
86
  "revision": "self",
87
- "artifact_fingerprint_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
88
  "metrics": {
89
  "audio.audio_to_text.recall@1": {
90
  "denominator": 2611,
@@ -157,7 +157,11 @@
157
  "value": 0.819650067294751
158
  }
159
  },
160
- "parameters": 144554112
 
 
 
 
161
  }
162
  },
163
  "datasets": {
@@ -475,10 +479,15 @@
475
  "nano": {
476
  "repo_id": "llm-semantic-router/Vela-1.0-Omni-Nano",
477
  "revision": "self",
478
- "native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
479
- "parameters": 144554112,
480
  "embedding_dimensions": 384,
481
- "max_text_tokens": 512
 
 
 
 
 
482
  },
483
  "mini": {
484
  "repo_id": "llm-semantic-router/Vela-1.0-Omni-Mini",
@@ -2624,14 +2633,14 @@
2624
  "tokens_include_special_tokens": true,
2625
  "public_text_overflow": "Raises ValueError; shorten or explicitly chunk longer inputs.",
2626
  "embedding_dimensions": 384,
2627
- "parameters": 144554112
2628
  },
2629
  "efficiency": {
2630
  "status": "not_measured_for_this_release",
2631
- "parameters": 144554112,
2632
  "embedding_dimensions": 384,
2633
  "dense_fp32_index_bytes_per_vector": 1536,
2634
- "weights_safetensors_bytes": 578289664,
2635
  "scope": "No new latency or peak-memory measurement is claimed. Vector storage excludes index structures and metadata.",
2636
  "historical_measurement": {
2637
  "attention": "native SDPA",
@@ -2669,6 +2678,14 @@
2669
  "warmup_batches": 2,
2670
  "status": "historical_previous_release_measurement",
2671
  "applicability": "Historical measurement on its recorded native artifact; not applicable to this text-backbone update."
 
 
 
 
 
 
 
 
2672
  }
2673
  },
2674
  "previous_release": {
@@ -3725,6 +3742,13 @@
3725
  "report": "benchmarks/component-equivalence.json",
3726
  "new_full_panel_inference": false,
3727
  "basis": "199 exact text tensors and byte-identical inference/tokenizer/configuration under the same official evaluator protocol."
 
 
 
 
 
 
 
3728
  }
3729
  },
3730
  {
@@ -3754,7 +3778,14 @@
3754
  "definition": "Official TaskResult aggregation within each task, followed by official mean_task, mean_task_type and task_types aggregation. The 134 subset/split slots are not averaged as independent tasks.",
3755
  "score_scale": "0-1; multiply by 100 for display"
3756
  },
3757
- "report": "benchmarks/maeb-audio-only.json"
 
 
 
 
 
 
 
3758
  }
3759
  ],
3760
  "net_benefit": {
@@ -4295,7 +4326,21 @@
4295
  "VoxPopuliGenderClustering": 0.0,
4296
  "VoxPopuliLanguageID": 0.0
4297
  },
4298
- "paired_interval_scope": "Paired intervals cover the original common metrics, family and two-primary-per-family utility. They are not confidence intervals for the new all-14 common mean or the four-component product aggregate."
 
 
 
 
 
 
 
4299
  },
4300
- "paired_uncertainty_scope": "Paired intervals cover the original common metrics, family and two-primary-per-family utility. They are not confidence intervals for the new all-14 common mean or the four-component product aggregate."
 
 
 
 
 
 
 
4301
  }
 
1
  {
2
  "schema_version": 2,
3
  "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
4
+ "revision_semantics": "self denotes this published inference package; evaluated_* identities retain original measurements. The exact public-inference equivalence report binds score applicability.",
5
  "score_scale": "0-1; multiply by 100 for percentages",
6
  "matrix": {
7
  "small": {
 
84
  "candidate": {
85
  "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
86
  "revision": "self",
87
+ "artifact_fingerprint_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
88
  "metrics": {
89
  "audio.audio_to_text.recall@1": {
90
  "denominator": 2611,
 
157
  "value": 0.819650067294751
158
  }
159
  },
160
+ "parameters": 135383808,
161
+ "evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
162
+ "evaluated_revision": "de02c367bc9aa7a6f7ad3e59fb38962502e021c1",
163
+ "evaluated_parameters": 144554112,
164
+ "inference_equivalence_report": "benchmarks/inference-equivalence.json"
165
  }
166
  },
167
  "datasets": {
 
479
  "nano": {
480
  "repo_id": "llm-semantic-router/Vela-1.0-Omni-Nano",
481
  "revision": "self",
482
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
483
+ "parameters": 135383808,
484
  "embedding_dimensions": 384,
485
+ "max_text_tokens": 512,
486
+ "evaluated_native_artifact_sha256": "96d76f97478e114b816df3ab1b3c6947e4c5ef0babfb40ab865d7e3c838d34c4",
487
+ "evaluated_revision": "de02c367bc9aa7a6f7ad3e59fb38962502e021c1",
488
+ "evaluated_parameters": 144554112,
489
+ "inference_equivalence_report": "benchmarks/inference-equivalence.json",
490
+ "revision_kind": "HF_snapshot"
491
  },
492
  "mini": {
493
  "repo_id": "llm-semantic-router/Vela-1.0-Omni-Mini",
 
2633
  "tokens_include_special_tokens": true,
2634
  "public_text_overflow": "Raises ValueError; shorten or explicitly chunk longer inputs.",
2635
  "embedding_dimensions": 384,
2636
+ "parameters": 135383808
2637
  },
2638
  "efficiency": {
2639
  "status": "not_measured_for_this_release",
2640
+ "parameters": 135383808,
2641
  "embedding_dimensions": 384,
2642
  "dense_fp32_index_bytes_per_vector": 1536,
2643
+ "weights_safetensors_bytes": 541598024,
2644
  "scope": "No new latency or peak-memory measurement is claimed. Vector storage excludes index structures and metadata.",
2645
  "historical_measurement": {
2646
  "attention": "native SDPA",
 
2678
  "warmup_batches": 2,
2679
  "status": "historical_previous_release_measurement",
2680
  "applicability": "Historical measurement on its recorded native artifact; not applicable to this text-backbone update."
2681
+ },
2682
+ "parameter_storage_change": {
2683
+ "old_safetensors_bytes": 578289664,
2684
+ "new_safetensors_bytes": 541598024,
2685
+ "reduction_bytes": 36691640,
2686
+ "old_FP32_parameter_bytes": 578216448,
2687
+ "new_FP32_parameter_bytes": 541535232,
2688
+ "scope": "Parameter storage only; not measured process peak memory or latency. Vector and index dimensions are unchanged."
2689
  }
2690
  },
2691
  "previous_release": {
 
3742
  "report": "benchmarks/component-equivalence.json",
3743
  "new_full_panel_inference": false,
3744
  "basis": "199 exact text tensors and byte-identical inference/tokenizer/configuration under the same official evaluator protocol."
3745
+ },
3746
+ "inference_profile_applicability": {
3747
+ "revision": "self",
3748
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
3749
+ "parameters": 135383808,
3750
+ "report": "benchmarks/inference-equivalence.json",
3751
+ "new_benchmark_run": false
3752
  }
3753
  },
3754
  {
 
3778
  "definition": "Official TaskResult aggregation within each task, followed by official mean_task, mean_task_type and task_types aggregation. The 134 subset/split slots are not averaged as independent tasks.",
3779
  "score_scale": "0-1; multiply by 100 for display"
3780
  },
3781
+ "report": "benchmarks/maeb-audio-only.json",
3782
+ "inference_profile_applicability": {
3783
+ "revision": "self",
3784
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
3785
+ "parameters": 135383808,
3786
+ "report": "benchmarks/inference-equivalence.json",
3787
+ "new_benchmark_run": false
3788
+ }
3789
  }
3790
  ],
3791
  "net_benefit": {
 
4326
  "VoxPopuliGenderClustering": 0.0,
4327
  "VoxPopuliLanguageID": 0.0
4328
  },
4329
+ "paired_interval_scope": "Paired intervals cover the original common metrics, family and two-primary-per-family utility. They are not confidence intervals for the new all-14 common mean or the four-component product aggregate.",
4330
+ "inference_profile_applicability": {
4331
+ "revision": "self",
4332
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
4333
+ "parameters": 135383808,
4334
+ "report": "benchmarks/inference-equivalence.json",
4335
+ "new_benchmark_run": false
4336
+ }
4337
  },
4338
+ "paired_uncertainty_scope": "Paired intervals cover the original common metrics, family and two-primary-per-family utility. They are not confidence intervals for the new all-14 common mean or the four-component product aggregate.",
4339
+ "inference_profile": {
4340
+ "revision": "self",
4341
+ "native_artifact_sha256": "50ce808197fb43a1b8913a66d4a47339ec66ecd10d52cbd9f9e752d02f8c43fa",
4342
+ "parameters": 135383808,
4343
+ "report": "benchmarks/inference-equivalence.json",
4344
+ "new_benchmark_run": false
4345
+ }
4346
  }
vela_omni.py CHANGED
@@ -10,13 +10,17 @@ from torch.nn import functional as F
10
 
11
  from omni_components.residual_projection import ResidualTextProjection, duration_mean
12
  from omni_components.embedder import MultimodalEmbedder
 
13
  from omni_components.mini import MultiModalSentenceEmbedder
14
 
15
 
16
  class VelaOmni(nn.Module):
17
- def __init__(self, root, variant, legacy=False, audio_projection="identity", text_projection="identity", audio_pooling="mean", max_text_length=None, text_pooling="mean"):
18
  super().__init__()
19
  self.variant = variant
 
 
 
20
  if text_projection not in ("identity", "residual_rank64"):
21
  raise ValueError("Unsupported text projection")
22
  if audio_pooling not in ("mean", "duration_mean") or (variant != "nano" and audio_pooling != "mean"):
@@ -35,7 +39,8 @@ class VelaOmni(nn.Module):
35
  raise ValueError(f"Unsupported {variant} max_text_length: {max_text_length}")
36
  self.max_text_length = max_text_length
37
  if variant == "nano":
38
- self.model = MultimodalEmbedder(Path(root) / "components", legacy=legacy,
 
39
  max_text_length=self.max_text_length, text_pooling=self.text_pooling)
40
  self.tokenizer = self.model.text_encoder.tokenizer
41
  if text_projection == "residual_rank64":
@@ -62,15 +67,20 @@ class VelaOmni(nn.Module):
62
  root = Path(snapshot_download(repo_id, revision=revision,
63
  allow_patterns=["config.json", "model.safetensors", "components/*"]))
64
  config = json.loads((root / "config.json").read_text())
65
- if config.get("format_version") not in (1, 2, 3, 4):
66
  raise ValueError("Unsupported Vela Omni artifact format")
 
 
 
 
67
  if config.get("image_pooling", "mean") != "mean":
68
  raise ValueError("Unsupported image pooling")
69
  instance = cls(root, config["variant"], audio_projection=config.get("audio_projection", "identity"),
70
  text_projection=config.get("text_projection", "identity"),
71
  audio_pooling=config.get("audio_pooling", "mean"),
72
  max_text_length=config.get("max_text_length"),
73
- text_pooling=config.get("text_pooling", "mean"))
 
74
  weights = load_file(str(root / "model.safetensors"), device="cpu")
75
  result = instance.model.load_state_dict(weights, strict=True, assign=True)
76
  if result.missing_keys or result.unexpected_keys:
 
10
 
11
  from omni_components.residual_projection import ResidualTextProjection, duration_mean
12
  from omni_components.embedder import MultimodalEmbedder
13
+ from omni_components.single_modality import SingleModalityEmbedder
14
  from omni_components.mini import MultiModalSentenceEmbedder
15
 
16
 
17
  class VelaOmni(nn.Module):
18
+ def __init__(self, root, variant, legacy=False, audio_projection="identity", text_projection="identity", audio_pooling="mean", max_text_length=None, text_pooling="mean", inference_profile="full"):
19
  super().__init__()
20
  self.variant = variant
21
+ if inference_profile not in ("full", "single_modality") or (inference_profile == "single_modality" and variant != "nano"):
22
+ raise ValueError("Unsupported inference profile")
23
+ self.inference_profile = inference_profile
24
  if text_projection not in ("identity", "residual_rank64"):
25
  raise ValueError("Unsupported text projection")
26
  if audio_pooling not in ("mean", "duration_mean") or (variant != "nano" and audio_pooling != "mean"):
 
39
  raise ValueError(f"Unsupported {variant} max_text_length: {max_text_length}")
40
  self.max_text_length = max_text_length
41
  if variant == "nano":
42
+ encoder_type = SingleModalityEmbedder if inference_profile == "single_modality" else MultimodalEmbedder
43
+ self.model = encoder_type(Path(root) / "components", legacy=legacy,
44
  max_text_length=self.max_text_length, text_pooling=self.text_pooling)
45
  self.tokenizer = self.model.text_encoder.tokenizer
46
  if text_projection == "residual_rank64":
 
67
  root = Path(snapshot_download(repo_id, revision=revision,
68
  allow_patterns=["config.json", "model.safetensors", "components/*"]))
69
  config = json.loads((root / "config.json").read_text())
70
+ if config.get("format_version") not in (1, 2, 3, 4, 5):
71
  raise ValueError("Unsupported Vela Omni artifact format")
72
+ if config.get("format_version") == 5 and (config.get("variant") != "nano" or config.get("inference_profile") != "single_modality"):
73
+ raise ValueError("Format 5 requires Nano single-modality profile")
74
+ if config.get("format_version") != 5 and config.get("inference_profile", "full") != "full":
75
+ raise ValueError("Inference profile requires format 5")
76
  if config.get("image_pooling", "mean") != "mean":
77
  raise ValueError("Unsupported image pooling")
78
  instance = cls(root, config["variant"], audio_projection=config.get("audio_projection", "identity"),
79
  text_projection=config.get("text_projection", "identity"),
80
  audio_pooling=config.get("audio_pooling", "mean"),
81
  max_text_length=config.get("max_text_length"),
82
+ text_pooling=config.get("text_pooling", "mean"),
83
+ inference_profile=config.get("inference_profile", "full"))
84
  weights = load_file(str(root / "model.safetensors"), device="cpu")
85
  result = instance.model.load_state_dict(weights, strict=True, assign=True)
86
  if result.missing_keys or result.unexpected_keys: