Improve multimodal embeddings

README.md

@@ -22,7 +22,7 @@ tags:
 
 # Vela Omni Nano
 
-Vela Omni Nano maps text, images, and speech into a shared embedding space for multimodal search and matching.
+Vela Omni Nano maps text, images, and speech into a shared embedding space for multimodal search, routing, and clustering.
 
 **134M parameters · 384 dimensions · L2-normalized embeddings.**
 
@@ -34,26 +34,26 @@ Scores use a 0–100 scale; higher is better. All applicable models use the same
 
 | Metric | [Vela-1.0-Encoder-307M-Embedding](https://huggingface.co/llm-semantic-router/Vela-1.0-Encoder-307M-Embedding) | [multi-modal-embed-small](https://huggingface.co/llm-semantic-router/multi-modal-embed-small) | [multi-modal-embed-large](https://huggingface.co/llm-semantic-router/multi-modal-embed-large) | [Vela-1.0-Omni-Nano](https://huggingface.co/llm-semantic-router/Vela-1.0-Omni-Nano) |
 |---|---:|---:|---:|---:|
-| Banking77 · Accuracy | 80.00 | 70.42 | 75.78 | 70.42 |
-| MASSIVE English · Accuracy | 75.64 | 65.95 | 72.31 | 65.95 |
-| COCO · Image → text · R@1 | N/A | 40.83 | 42.53 | 44.23 |
-| COCO · Image → text · R@5 | N/A | 67.19 | 75.21 | 73.03 |
-| COCO · Image → text · R@10 | N/A | 78.49 | 87.61 | 86.03 |
-| COCO · Text → image · R@1 | N/A | 30.18 | 35.04 | 36.11 |
-| COCO · Text → image · R@5 | N/A | 59.42 | 70.09 | 68.72 |
-| COCO · Text → image · R@10 | N/A | 74.29 | 83.91 | 82.02 |
-| LibriSpeech · Audio → text · R@1 | N/A | 4.21 | 56.99 | 5.29 |
-| LibriSpeech · Audio → text · R@5 | N/A | 11.99 | 81.85 | 15.82 |
-| LibriSpeech · Audio → text · R@10 | N/A | 19.03 | 87.94 | 23.90 |
-| LibriSpeech · Text → audio · R@1 | N/A | 9.58 | 78.58 | 11.61 |
-| LibriSpeech · Text → audio · R@5 | N/A | 22.53 | 94.02 | 26.55 |
-| LibriSpeech · Text → audio · R@10 | N/A | 30.69 | 97.01 | 35.44 |
-
-Text evaluation uses fixed class prototypes: 3,080 Banking77 and 2,972 MASSIVE English queries. This is nearest-prototype classification, not the MTEB classification protocol.
+| Banking77 · Accuracy | 80.00 | 70.42 | 75.78 | 73.28 |
+| MASSIVE English · Accuracy | 75.64 | 65.95 | 72.31 | 66.55 |
+| COCO · Image → text · R@1 | N/A | 40.83 | 42.53 | 47.02 |
+| COCO · Image → text · R@5 | N/A | 67.19 | 75.21 | 74.97 |
+| COCO · Image → text · R@10 | N/A | 78.49 | 87.61 | 85.91 |
+| COCO · Text → image · R@1 | N/A | 30.18 | 35.04 | 37.33 |
+| COCO · Text → image · R@5 | N/A | 59.42 | 70.09 | 70.16 |
+| COCO · Text → image · R@10 | N/A | 74.29 | 83.91 | 82.99 |
+| LibriSpeech · Audio → text · R@1 | N/A | 4.21 | 56.99 | 8.20 |
+| LibriSpeech · Audio → text · R@5 | N/A | 11.99 | 81.85 | 21.29 |
+| LibriSpeech · Audio → text · R@10 | N/A | 19.03 | 87.94 | 30.26 |
+| LibriSpeech · Text → audio · R@1 | N/A | 9.58 | 78.58 | 13.26 |
+| LibriSpeech · Text → audio · R@5 | N/A | 22.53 | 94.02 | 30.00 |
+| LibriSpeech · Text → audio · R@10 | N/A | 30.69 | 97.01 | 38.66 |
+
+Text evaluation uses fixed class prototypes: 3,080 Banking77 and 2,972 MASSIVE English queries. Vela Omni is adapted using training examples and intent labels from these two datasets; comparison models are evaluated as published without additional adaptation here. These scores measure supervised task adaptation with nearest-prototype classification, not zero-shot transfer or the MTEB classification protocol.
 
 Image retrieval uses the COCO Karpathy CC-BY2 image subset: 823 images and 4,115 captions. Speech retrieval uses 2,611 LibriSpeech test-clean clips and 2,610 unique transcripts, with clips limited to 30 seconds. Recall is measured against each complete candidate pool, accepting all matching positives.
 
-Evaluation uses FP32 embeddings, cosine similarity, and a common 128-token text limit. [Full scores and protocol](./scores.json) include model revisions, sample counts, and paired uncertainty.
+Evaluation uses FP32 embeddings, cosine similarity, and a common 128-token text limit. [Full scores and protocol](./scores.json) include model revisions, sample counts, and paired uncertainty against the original and previous models. These fixed evaluation pools are reused across releases.
 
 ## Quick start
 
@@ -72,10 +72,10 @@ vectors = model.encode_text(["A bicycle beside a building.", "Someone is reading
 print(vectors.shape)  # (2, 384)
 ```
 
-Pass a list of Pillow images to `model.encode_image(images)`. Pass a list of mono NumPy waveforms to `model.encode_audio(waveforms, sampling_rate=16000)`; each waveform must be at most 30 seconds. Compare normalized vectors with their dot product. Similarity scores are rankings, not calibrated probabilities.
+Pass a list of Pillow images to `model.encode_image(images)`. Pass a list of mono NumPy waveforms to `model.encode_audio(waveforms, sampling_rate=16000)`; each waveform must be at most 30 seconds. Compare normalized vectors with their dot product. To route media, embed each destination’s name and description, then select the closest vector. To discover routing categories, cluster media embeddings and inspect each group. Similarity scores are rankings, not calibrated probabilities.
 
-Cross-modal alignment uses COCO image–caption pairs from the CC-BY 2.0 image subset and [LibriSpeech](https://www.openslr.org/12/) speech–transcript pairs (CC-BY 4.0). COCO annotations are provided under CC-BY 4.0.
+Cross-modal alignment uses COCO image–caption pairs from the CC-BY 2.0 image subset and [LibriSpeech](https://www.openslr.org/12/) speech–transcript pairs (CC-BY 4.0). COCO annotations, Banking77, and MASSIVE are provided under CC-BY 4.0.
 
 The repository includes the native model code, component configurations, and tokenizer and processor files. See [NOTICE](./NOTICE) and [LICENSE](./LICENSE) for component attribution and license terms.
 
-[Explore the Vela model collection](https://huggingface.co/collections/llm-semantic-router/vela-10-router-models-6aa555ba70cc6997d6d67798)
+[Explore the Vela model collection](https://huggingface.co/collections/llm-semantic-router/vela-10-6aa555ba70cc6997d6d67798)

config.json

@@ -1,7 +1,7 @@
 {
-  "format_version": 2,
+  "format_version": 3,
   "variant": "nano",
-  "parameter_count": 133907328,
+  "parameter_count": 133956480,
   "embedding_dim": 384,
   "max_text_length": 128,
   "architectures": [
@@ -11,5 +11,7 @@
   "library_name": "pytorch",
   "audio_sampling_rate": 16000,
   "audio_max_seconds": 30,
-  "audio_projection": "linear"
+  "audio_projection": "linear",
+  "text_projection": "residual_rank64",
+  "audio_pooling": "mean"
 }

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:aa28600e25bdfb3796fc173fd9504c5a4687c07211d7e3dbb751dceef1a8c341
-size 535689632
+oid sha256:949b21264db9d963de79a56dd2aeb4f1f31dce315dc54ee15296b7c69772e7b9
+size 535886448

omni_components/residual_projection.py

@@ -0,0 +1,34 @@
+"""Native rank-64 residual text mapping and duration-aware audio readout."""
+import torch
+from torch import nn
+
+
+class ResidualTextProjection(nn.Module):
+    def __init__(self, dimension, seed=20260919):
+        super().__init__()
+        self.V = nn.Linear(dimension, 64, bias=False)
+        self.U = nn.Linear(64, dimension, bias=False)
+        generator = torch.Generator(device='cpu').manual_seed(seed)
+        with torch.no_grad():
+            self.V.weight.copy_(torch.randn(64, dimension, generator=generator) * 0.01)
+            self.U.weight.zero_()
+
+    def forward(self, values):
+        return values + self.U(self.V(values))
+
+
+def duration_frame_counts(sample_counts, frame_count, device=None):
+    counts = torch.as_tensor(sample_counts, device=device)
+    if counts.ndim != 1 or counts.dtype not in (torch.int32, torch.int64):
+        raise ValueError('Original integer sample counts are required')
+    if not bool(((counts > 0) & (counts <= 480000)).all()) or frame_count != 1500:
+        raise ValueError('Expected valid 16kHz clips and native 1500-frame output')
+    return torch.clamp(torch.div(counts + 319, 320, rounding_mode='floor'), 1, frame_count)
+
+
+def duration_mean(hidden_states, sample_counts):
+    if hidden_states.ndim != 3 or len(sample_counts) != len(hidden_states):
+        raise ValueError('Audio lengths do not match encoder batch')
+    counts = duration_frame_counts(sample_counts, hidden_states.shape[1], hidden_states.device)
+    mask = torch.arange(hidden_states.shape[1], device=hidden_states.device)[None, :] < counts[:, None]
+    return (hidden_states * mask[:, :, None]).sum(dim=1) / counts[:, None].to(hidden_states.dtype)

scores.json

@@ -1,5 +1,5 @@
 {
-  "schema_version": 1,
+  "schema_version": 2,
   "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
   "revision_semantics": "self denotes the Hugging Face snapshot containing this file; the native artifact fingerprint binds its evaluated model files.",
   "score_scale": "0-1; multiply by 100 for percentages",
@@ -182,77 +182,77 @@
     "candidate": {
       "model": "llm-semantic-router/Vela-1.0-Omni-Nano",
       "revision": "self",
-      "artifact_fingerprint_sha256": "32f410c3febdbd02e3f790b907db482e4ae1d493e6a1c4eaab375ebada234f8b",
+      "artifact_fingerprint_sha256": "e4c8165db4e76534f71adbe24719d464b3d416ee9e7fe49c52a5aa05b3045371",
       "metrics": {
         "text.banking77.accuracy": {
-          "numerator": 2169,
+          "numerator": 2257,
           "denominator": 3080,
-          "value": 0.7042207792207792
+          "value": 0.7327922077922078
         },
         "text.massive-en.accuracy": {
-          "numerator": 1960,
+          "numerator": 1978,
           "denominator": 2972,
-          "value": 0.6594885598923284
+          "value": 0.6655450874831763
         },
         "image.image_to_text.recall@1": {
-          "numerator": 364,
+          "numerator": 387,
           "denominator": 823,
-          "value": 0.4422843256379101
+          "value": 0.47023086269744835
         },
         "image.image_to_text.recall@5": {
-          "numerator": 601,
+          "numerator": 617,
           "denominator": 823,
-          "value": 0.7302551640340219
+          "value": 0.7496962332928311
         },
         "image.image_to_text.recall@10": {
-          "numerator": 708,
+          "numerator": 707,
           "denominator": 823,
-          "value": 0.8602673147023087
+          "value": 0.8590522478736331
         },
         "image.text_to_image.recall@1": {
-          "numerator": 1486,
+          "numerator": 1536,
           "denominator": 4115,
-          "value": 0.3611178614823815
+          "value": 0.3732685297691373
         },
         "image.text_to_image.recall@5": {
-          "numerator": 2828,
+          "numerator": 2887,
           "denominator": 4115,
-          "value": 0.6872417982989064
+          "value": 0.7015795868772783
         },
         "image.text_to_image.recall@10": {
-          "numerator": 3375,
+          "numerator": 3415,
           "denominator": 4115,
-          "value": 0.8201701093560145
+          "value": 0.8298906439854192
         },
         "audio.audio_to_text.recall@1": {
-          "numerator": 138,
+          "numerator": 214,
           "denominator": 2611,
-          "value": 0.05285331290693221
+          "value": 0.0819609345078514
         },
         "audio.audio_to_text.recall@5": {
-          "numerator": 413,
+          "numerator": 556,
           "denominator": 2611,
-          "value": 0.1581769436997319
+          "value": 0.21294523171198773
         },
         "audio.audio_to_text.recall@10": {
-          "numerator": 624,
+          "numerator": 790,
           "denominator": 2611,
-          "value": 0.23898889314438912
+          "value": 0.30256606664113367
         },
         "audio.text_to_audio.recall@1": {
-          "numerator": 303,
+          "numerator": 346,
           "denominator": 2610,
-          "value": 0.11609195402298851
+          "value": 0.13256704980842912
         },
         "audio.text_to_audio.recall@5": {
-          "numerator": 693,
+          "numerator": 783,
           "denominator": 2610,
-          "value": 0.2655172413793103
+          "value": 0.3
         },
         "audio.text_to_audio.recall@10": {
-          "numerator": 925,
+          "numerator": 1009,
           "denominator": 2610,
-          "value": 0.3544061302681992
+          "value": 0.38659003831417627
         }
       }
     }
@@ -312,109 +312,221 @@
     "similarity": "FP32 cosine of L2-normalized embeddings",
     "ties": "lexicographic candidate ID",
     "retrieval": "complete candidate pools; all annotated positives",
-    "classification": "nearest normalized mean class prototype; fixed prototype split",
+    "classification": "nearest normalized mean class prototype; Vela Omni uses supervised training-split intent-label adaptation; comparison models evaluated as published",
     "aggregation": "Each family is the mean of its two primary metrics; utility is the mean of all six primaries. Retrieval primaries are R@1; R@5 and R@10 are reported separately.",
-    "evaluation_scope": "fixed test-split evaluation pools"
+    "evaluation_scope": "fixed test-split evaluation pools reused across releases"
   },
   "paired_uncertainty": {
-    "comparison": {
-      "candidate": "llm-semantic-router/Vela-1.0-Omni-Nano",
-      "baseline": "llm-semantic-router/multi-modal-embed-small"
-    },
-    "seed": 20260918,
-    "replicates": 2000,
-    "method": "paired stratified group percentile; PCG64; fixed candidate pools",
-    "intervals": {
-      "audio.audio_to_text.recall@1": {
-        "point_delta": 0.010723860589812333,
-        "lower": 0.005401297417205939,
-        "upper": 0.016930329895273797
-      },
-      "audio.audio_to_text.recall@10": {
-        "point_delta": 0.048640367675220224,
-        "lower": 0.03862647576747038,
-        "upper": 0.057686079679882395
-      },
-      "audio.audio_to_text.recall@5": {
-        "point_delta": 0.038299502106472615,
-        "lower": 0.028966882934863172,
-        "upper": 0.04718931610578753
-      },
-      "audio.text_to_audio.recall@1": {
-        "point_delta": 0.020306513409961684,
-        "lower": 0.01207735759631001,
-        "upper": 0.029131333167300467
-      },
-      "audio.text_to_audio.recall@10": {
-        "point_delta": 0.0475095785440613,
-        "lower": 0.036695309471507946,
-        "upper": 0.05838249564916254
-      },
-      "audio.text_to_audio.recall@5": {
-        "point_delta": 0.040229885057471264,
-        "lower": 0.027551627678505008,
-        "upper": 0.05248271039052227
-      },
-      "family.audio": {
-        "point_delta": 0.015515186999887009,
-        "lower": 0.010126625706639377,
-        "upper": 0.021365285246569197
+    "original": {
+      "comparison": {
+        "candidate": "llm-semantic-router/Vela-1.0-Omni-Nano",
+        "baseline": "llm-semantic-router/multi-modal-embed-small",
+        "baseline_revision": "fdf8e01b7b0f3a69ac1ac8e2a64dcb1ede177ba4",
+        "baseline_artifact_fingerprint_sha256": "684fe0c48a4ddd3944167f6e5dc75e97ada24e5b120bd9a2921f00dc03112eeb"
       },
-      "family.image": {
-        "point_delta": 0.04665856622114216,
-        "lower": 0.031831713244228434,
-        "upper": 0.061849939246658554
-      },
-      "family.text": {
-        "point_delta": 0.0,
-        "lower": 0.0,
-        "upper": 0.0
-      },
-      "image.image_to_text.recall@1": {
-        "point_delta": 0.03402187120291616,
-        "lower": 0.009720534629404616,
-        "upper": 0.05953827460510328
-      },
-      "image.image_to_text.recall@10": {
-        "point_delta": 0.07533414337788578,
-        "lower": 0.05103280680437424,
-        "upper": 0.10085054678007291
-      },
-      "image.image_to_text.recall@5": {
-        "point_delta": 0.0583232077764277,
-        "lower": 0.031591737545565005,
-        "upper": 0.08383961117861483
-      },
-      "image.text_to_image.recall@1": {
-        "point_delta": 0.059295261239368166,
-        "lower": 0.04325637910085055,
-        "upper": 0.07630619684082625
-      },
-      "image.text_to_image.recall@10": {
-        "point_delta": 0.07727825030376671,
-        "lower": 0.061482381530984204,
-        "upper": 0.0928311057108141
-      },
-      "image.text_to_image.recall@5": {
-        "point_delta": 0.09307411907654921,
-        "lower": 0.07606318347509113,
-        "upper": 0.11009113001215064
-      },
-      "text.banking77.accuracy": {
-        "point_delta": 0.0,
-        "lower": 0.0,
-        "upper": 0.0
-      },
-      "text.massive-en.accuracy": {
-        "point_delta": 0.0,
-        "lower": 0.0,
-        "upper": 0.0
+      "seed": 20260918,
+      "replicates": 2000,
+      "method": "paired stratified group percentile; PCG64; fixed candidate pools",
+      "intervals": {
+        "audio.audio_to_text.recall@1": {
+          "point_delta": 0.03983148219073152,
+          "lower": 0.0325962321534125,
+          "upper": 0.047758993112907225
+        },
+        "audio.audio_to_text.recall@10": {
+          "point_delta": 0.11221754117196477,
+          "lower": 0.09530973319687737,
+          "upper": 0.12877810022670813
+        },
+        "audio.audio_to_text.recall@5": {
+          "point_delta": 0.09306779011872846,
+          "lower": 0.07882957120502015,
+          "upper": 0.10728804617676266
+        },
+        "audio.text_to_audio.recall@1": {
+          "point_delta": 0.0367816091954023,
+          "lower": 0.026758054192447193,
+          "upper": 0.04703357251126449
+        },
+        "audio.text_to_audio.recall@10": {
+          "point_delta": 0.07969348659003832,
+          "lower": 0.06278676122484814,
+          "upper": 0.09535034581998793
+        },
+        "audio.text_to_audio.recall@5": {
+          "point_delta": 0.07471264367816093,
+          "lower": 0.058304011796927804,
+          "upper": 0.08890972870306472
+        },
+        "family.audio": {
+          "point_delta": 0.03830654569306691,
+          "lower": 0.0317536163235943,
+          "upper": 0.04491423341337552
+        },
+        "family.image": {
+          "point_delta": 0.06670716889428918,
+          "lower": 0.04933171324422843,
+          "upper": 0.08457168894289184
+        },
+        "family.text": {
+          "point_delta": 0.017313978081138243,
+          "lower": 0.012190932632665139,
+          "upper": 0.022698066843763527
+        },
+        "image.image_to_text.recall@1": {
+          "point_delta": 0.061968408262454436,
+          "lower": 0.03402187120291616,
+          "upper": 0.09234507897934387
+        },
+        "image.image_to_text.recall@10": {
+          "point_delta": 0.07411907654921021,
+          "lower": 0.04738760631834751,
+          "upper": 0.10085054678007291
+        },
+        "image.image_to_text.recall@5": {
+          "point_delta": 0.07776427703523693,
+          "lower": 0.04981773997569866,
+          "upper": 0.10814094775212636
+        },
+        "image.text_to_image.recall@1": {
+          "point_delta": 0.07144592952612394,
+          "lower": 0.05297691373025516,
+          "upper": 0.09015795868772783
+        },
+        "image.text_to_image.recall@10": {
+          "point_delta": 0.08699878493317133,
+          "lower": 0.07047387606318348,
+          "upper": 0.10352369380315918
+        },
+        "image.text_to_image.recall@5": {
+          "point_delta": 0.10741190765492102,
+          "lower": 0.08894289185905224,
+          "upper": 0.12613001215066824
+        },
+        "text.banking77.accuracy": {
+          "point_delta": 0.02857142857142857,
+          "lower": 0.020461188697629102,
+          "upper": 0.03666480107400848
+        },
+        "text.massive-en.accuracy": {
+          "point_delta": 0.006056527590847914,
+          "lower": -0.00033658984096547205,
+          "upper": 0.013131423701480958
+        },
+        "utility": {
+          "point_delta": 0.04077589755616478,
+          "lower": 0.034528310785056776,
+          "upper": 0.047151560938567774
+        }
+      }
+    },
+    "previous": {
+      "comparison": {
+        "candidate": "llm-semantic-router/Vela-1.0-Omni-Nano",
+        "baseline": "llm-semantic-router/Vela-1.0-Omni-Nano",
+        "baseline_revision": "82f31e22b31f01989686559d974159034fa8118c",
+        "baseline_artifact_fingerprint_sha256": "32f410c3febdbd02e3f790b907db482e4ae1d493e6a1c4eaab375ebada234f8b"
       },
-      "utility": {
-        "point_delta": 0.020724584407009722,
-        "lower": 0.015571272805177982,
-        "upper": 0.026256156724631264
+      "seed": 20260918,
+      "replicates": 2000,
+      "method": "paired stratified group percentile; PCG64; fixed candidate pools",
+      "intervals": {
+        "audio.audio_to_text.recall@1": {
+          "point_delta": 0.02910762160091919,
+          "lower": 0.021861939983398526,
+          "upper": 0.036651355202609114
+        },
+        "audio.audio_to_text.recall@10": {
+          "point_delta": 0.06357717349674455,
+          "lower": 0.05113845782378832,
+          "upper": 0.0757709413136462
+        },
+        "audio.audio_to_text.recall@5": {
+          "point_delta": 0.05476828801225584,
+          "lower": 0.04306058450795293,
+          "upper": 0.06643108320584735
+        },
+        "audio.text_to_audio.recall@1": {
+          "point_delta": 0.016475095785440614,
+          "lower": 0.008695160152854542,
+          "upper": 0.023524944652131282
+        },
+        "audio.text_to_audio.recall@10": {
+          "point_delta": 0.03218390804597701,
+          "lower": 0.02000716134837118,
+          "upper": 0.04482972738447
+        },
+        "audio.text_to_audio.recall@5": {
+          "point_delta": 0.034482758620689655,
+          "lower": 0.02224784413518575,
+          "upper": 0.04650418024307072
+        },
+        "family.audio": {
+          "point_delta": 0.0227913586931799,
+          "lower": 0.017447187890587527,
+          "upper": 0.028299187706426518
+        },
+        "family.image": {
+          "point_delta": 0.020048602673147023,
+          "lower": 0.009477521263669502,
+          "upper": 0.03062272174969622
+        },
+        "family.text": {
+          "point_delta": 0.017313978081138243,
+          "lower": 0.012190932632665139,
+          "upper": 0.022698066843763527
+        },
+        "image.image_to_text.recall@1": {
+          "point_delta": 0.027946537059538274,
+          "lower": 0.00850546780072904,
+          "upper": 0.04738760631834751
+        },
+        "image.image_to_text.recall@10": {
+          "point_delta": -0.001215066828675577,
+          "lower": -0.013365735115431349,
+          "upper": 0.00975091130012134
+        },
+        "image.image_to_text.recall@5": {
+          "point_delta": 0.019441069258809233,
+          "lower": 0.002430133657351154,
+          "upper": 0.03766707168894289
+        },
+        "image.text_to_image.recall@1": {
+          "point_delta": 0.012150668286755772,
+          "lower": 0.003639125151883354,
+          "upper": 0.020656136087484813
+        },
+        "image.text_to_image.recall@10": {
+          "point_delta": 0.009720534629404616,
+          "lower": 0.003888213851761847,
+          "upper": 0.015795868772782502
+        },
+        "image.text_to_image.recall@5": {
+          "point_delta": 0.014337788578371811,
+          "lower": 0.006804374240583232,
+          "upper": 0.02235722964763062
+        },
+        "text.banking77.accuracy": {
+          "point_delta": 0.02857142857142857,
+          "lower": 0.020461188697629102,
+          "upper": 0.03666480107400848
+        },
+        "text.massive-en.accuracy": {
+          "point_delta": 0.006056527590847914,
+          "lower": -0.00033658984096547205,
+          "upper": 0.013131423701480958
+        },
+        "utility": {
+          "point_delta": 0.020051313149155054,
+          "lower": 0.01592880942853571,
+          "upper": 0.024331405417179264
+        }
       }
     }
+  },
+  "training_scope": {
+    "text": "Banking77 and MASSIVE English training examples and intent labels; task-adapted classification, not zero-shot transfer or standard MTEB classification",
+    "image": "COCO CC-BY2 training image-caption subset",
+    "audio": "LibriSpeech train-clean-100 speech-transcript subset"
   }
 }

vela_omni.py

@@ -8,19 +8,27 @@ from safetensors.torch import load_file
 from torch import nn
 from torch.nn import functional as F
 
+from omni_components.residual_projection import ResidualTextProjection, duration_mean
 from omni_components.embedder import MultimodalEmbedder
 from omni_components.mini import MultiModalSentenceEmbedder
 
 
 class VelaOmni(nn.Module):
-    def __init__(self, root, variant, legacy=False, audio_projection="identity"):
+    def __init__(self, root, variant, legacy=False, audio_projection="identity", text_projection="identity", audio_pooling="mean"):
         super().__init__()
         self.variant = variant
+        if text_projection not in ("identity", "residual_rank64"):
+            raise ValueError("Unsupported text projection")
+        if audio_pooling not in ("mean", "duration_mean") or (variant != "nano" and audio_pooling != "mean"):
+            raise ValueError("Unsupported audio pooling")
+        self.audio_pooling = audio_pooling
         self.embedding_dim = 384 if variant == "nano" else 768
         self.max_text_length = 128 if variant == "nano" else 32768
         if variant == "nano":
             self.model = MultimodalEmbedder(Path(root) / "components", legacy=legacy)
             self.tokenizer = self.model.text_encoder.tokenizer
+            if text_projection == "residual_rank64":
+                self.model.text_encoder.projection = ResidualTextProjection(384)
             if audio_projection == "linear":
                 self.model.audio_encoder.projection = nn.Linear(384, 384)
             elif audio_projection != "identity":
@@ -30,6 +38,8 @@ class VelaOmni(nn.Module):
                 raise ValueError("Mini uses its native audio projection")
             self.model = MultiModalSentenceEmbedder(Path(root) / "components", legacy=legacy)
             self.tokenizer = self.model.text_model.tokenizer
+            if text_projection == "residual_rank64":
+                self.model.text_proj = ResidualTextProjection(768)
         else:
             raise ValueError("Unknown Vela Omni variant")
 
@@ -41,9 +51,11 @@ class VelaOmni(nn.Module):
             root = Path(snapshot_download(repo_id, revision=revision,
                     allow_patterns=["config.json", "model.safetensors", "components/*"]))
         config = json.loads((root / "config.json").read_text())
-        if config.get("format_version") not in (1, 2):
+        if config.get("format_version") not in (1, 2, 3):
             raise ValueError("Unsupported Vela Omni artifact format")
-        instance = cls(root, config["variant"], audio_projection=config.get("audio_projection", "identity"))
+        instance = cls(root, config["variant"], audio_projection=config.get("audio_projection", "identity"),
+                       text_projection=config.get("text_projection", "identity"),
+                       audio_pooling=config.get("audio_pooling", "mean"))
         weights = load_file(str(root / "model.safetensors"), device="cpu")
         result = instance.model.load_state_dict(weights, strict=True, assign=True)
         if result.missing_keys or result.unexpected_keys:
@@ -93,7 +105,14 @@ class VelaOmni(nn.Module):
         if any(len(w) > 30 * sampling_rate for w in waveforms):
             raise ValueError("Audio inputs must be at most 30 seconds")
         if self.variant == "nano":
-            values = self.model.encode_audio(waveforms, sampling_rate=sampling_rate)
+            if self.audio_pooling == "duration_mean":
+                encoder = self.model.audio_encoder
+                features = encoder.feature_extractor(waveforms, sampling_rate=sampling_rate,
+                    return_tensors="pt")["input_features"].to(next(encoder.parameters()))
+                hidden = encoder.encoder(input_features=features, output_hidden_states=False).last_hidden_state
+                values = encoder.projection(duration_mean(hidden, [len(w) for w in waveforms]))
+            else:
+                values = self.model.encode_audio(waveforms, sampling_rate=sampling_rate)
         else:
             features = self.model.audio_processor(waveforms, sampling_rate=sampling_rate,
                                     return_tensors="pt")["input_features"]