nomic-embed-text-v2-moe — ONNX

ONNX export of nomic-ai/nomic-embed-text-v2-moe for in-browser / on-device inference with Transformers.js and onnxruntime. To our knowledge this is the first ONNX build of this model.

All weights, tokenizer, and behavior are the upstream model's — this repo only changes the runtime format. See Numerical differences below for the exact cost of each variant.

The base model is a Mixture-of-Experts (8 experts, top-2) and could not be exported by optimum as-is: the expert dispatch (.tolist() + data-dependent control flow) and the rotary cache are untraceable. This export replaces the dispatch with an equivalent dense two-GEMM formulation and freezes the rotary cache; fp32 output is identical to the original (max |Δ| on last_hidden_state = 1.9e-05, pooled cosine 1.000000).

Usage (Transformers.js)

nomic-v2 is an asymmetric, instruction-tuned retriever: documents and queries take different prefixes. The prefixes are not optional — dropping them is a silent ~20-point recall drop, not an error.

import { pipeline } from '@huggingface/transformers';

const extractor = await pipeline('feature-extraction', '<ORG>/nomic-embed-text-v2-moe-ONNX', {
  dtype: 'fp16',   // fp16 -> onnx/model_fp16.onnx (recommended -- see "Batch-composition sensitivity" below)
});

const doc   = await extractor('search_document: Hemoglobin A1c in Blood', { pooling: 'mean', normalize: true });
const query = await extractor('search_query: hgb a1c',                     { pooling: 'mean', normalize: true });
// cosine(query, doc) — inputs are already L2-normalised, so it's a dot product.

Available files

`dtype`	file	size	notes
`fp32`	`onnx/model.onnx`	1814 MB	exact; reference
`fp16`	`onnx/model_fp16.onnx`	907 MB	recommended — no batch/platform-dependence, ~0.9998 cosine to fp32
`q8`	`onnx/model_quantized.onnx`	456 MB	signed per-channel int8 (dynamic) — smallest well-behaved option, see caveat below
`q4`	`onnx/model_q4.onnx`	353 MB	4-bit MatMul + int8 embedding

The graph exposes two outputs: last_hidden_state (Transformers.js mean-pools + normalises in JS) and a convenience sentence_embedding (mean-pooled, L2-normalised) for use directly from onnxruntime.

Batch-composition sensitivity (int8/uint8/q4) and the fp16 alternative

Every int8/uint8/q4 variant here uses dynamic activation quantization (onnxruntime.quantization.quantize_dynamic, the standard Transformers.js conversion path): activation scales are recomputed from each call's actual batch at runtime, so the same string embeds slightly differently depending on what else was in its batch — measured ~0.994 cosine to itself, the same order of magnitude as q8's ~0.995 fidelity to fp32.

That alone would be unremarkable rounding noise for most encoders. It is not here: this model has real top-2 MoE routing (6 TopK router gates), and that per-batch noise is enough to occasionally flip which expert(s) a token routes through. The consequence is CPU-microarchitecture- and execution-provider- dependent, because different kernels compute that per-batch scale slightly differently:

measured on	batch-composition cosine (same string, alone vs. batched)
ARM CPU	~0.994
x86_64 CPU	0.875–0.960 (min diagonal cosine across a small mixed-length probe)
CoreML (ANE/GPU) vs. CPU, same host	mean 0.9357, min 0.8996

fp16 has no per-batch scale to compute at all — every value's precision loss is fixed and local, with no discrete quantization level for the router's TopK to land on either side of depending on batch composition or which kernel ran it. Measured: batch-composition cosine 1.000000 on both ARM and x86_64 (native and emulated), and CoreML vs. CPU mean/min cosine both 1.0000 — the same host where q8 measured 0.9357/0.8996. It is not a speed win (CPU has no fast fp16 kernels for most ops, so it is not faster than fp32; CoreML remains ~4.5x slower than CPU here regardless of dtype, from an unrelated graph-fragmentation issue — see below), only a correctness and size one.

If you need reproducible output across runs, batches, hosts, or execution providers, use fp16. q8/uint8/q4 remain reasonable choices when you control both sides of a comparison (e.g. an index and its queries encoded by the same graph, same host, same execution provider) and the small non-determinism is acceptable.

CoreML is otherwise not recommended regardless of dtype: rotary_emb's Slice ops produce zero-sized dimensions CoreML does not support, fragmenting the graph (CoreML supports 715 of 1546 nodes on the fp16 graph, 831 of 1703 on the int8 graph, in ~120-130 partitions either way) and making it ~4.5x slower than CPU here regardless of which dtype you pick.

Numerical differences vs. the original

Measured against the fp32 graph on biomedical-label probes; retrieval on the BioSamples benchmark (151,174 ontology labels indexed, 3,340 messy queries), with both index and query encoded by the same graph.

variant	cosine to fp32 (mean)	R@1	R@5	R@10
fp32	1.0000	0.407	0.599	0.668
fp16	0.9998	*	*	*
int8 per-channel (`q8`)	0.9950	0.383	0.598	0.663
q4	0.9612	—	—	—
uint8 per-tensor	0.9193	0.382	0.607	0.676

* fp16 was not separately re-run against the BioSamples benchmark; a smaller retrieval-shaped harmonization test (109-term dictionary, 90 queries) gave identical auto-accept counts to fp32 exactly, where int8 shifted the count.

Takeaway on q8/q4/uint8: cosine-to-fp32 overstates the damage. uint8 is 8 cosine points below int8 yet ties it in retrieval (it even edges ahead at R@5/R@10) — the quantization error is largely a systematic rotation that cancels when index and query share it. The real cost of quantizing at all is ~2.5 R@1, the same for both. int8 earns a size-for-quality recommendation only under encoder skew (an fp32 index built server-side + a quantized query in the browser), where the error no longer cancels: there int8 costs 0.4 R@1 and uint8 costs 1.1, and uint8 gives up the most on the out-of-dictionary tail. If you quantize the index too, the choice stops mattering for retrieval quality -- it still carries the batch-composition caveat above regardless.