README.md

---
license: apache-2.0
base_model: google/siglip2-so400m-patch14-384
tags:
  - coreml
  - siglip2
  - image-encoder
  - apple-silicon
---

# SigLIP2-so400m image encoder, Core ML

The image tower of SigLIP2-so400m converted to Core ML (fp16), so it can run
with hardware acceleration on Macs. The weights are Google's, untouched — only
the format changed.

Converted from [`open_clip`](https://github.com/mlfoundations/open_clip),
model `ViT-SO400M-14-SigLIP2-378`, pretrained tag `webli`.

This is the **image tower only**. For image↔text search you also need the
matching text tower, which is not in this repository.

## Interface

| | |
|---|---|
| Input | `image`, an image of **378 × 378** |
| Output | `embedding`, **1152** dimensions, **float16**, already L2-normalized |
| Size | 815 MB |
| Minimum target | macOS 14 |

Two things are easy to get wrong, and neither one fails loudly — you just get
worse search results:

**The input is 378, not 384.** The original model is named `patch14-384`, but
its patch embedding is a 14×14 convolution with stride 14: 27 patches fit, and
the last 6 pixels of each dimension are dropped. So the model effectively looks
at the top-left 378×378 region of a 384-resized image, which is also what it
saw during training. If you resize your image to 378 directly instead of
resizing to 384 and cropping to 378, cosine similarity against the original
model drops from 0.998 to 0.970.

**The output is float16.** Reading the `MLMultiArray` buffer as float32 gives
you a plausible-looking vector that has nothing to do with the real embedding —
we measured 0.02 cosine similarity against the reference. Check `dataType`
rather than assuming.

Normalization is baked into the model (scale 2/255, bias −1, i.e. SigLIP2's
mapping to [−1, 1]), so pass raw pixels; do not normalize them yourself.

## Fidelity, and which compute unit to use

Measured on an M2 Pro against the fp16 ONNX export of the same model
([onnx-community/siglip2-so400m-patch14-384-ONNX](https://huggingface.co/onnx-community/siglip2-so400m-patch14-384-ONNX)),
over 134 frames drawn from 30 different videos, with identical preprocessing on
both sides. "Top-10 overlap" is how much the ranking of those 134 frames agrees
with the ONNX ranking across 30 real search queries — which is what a user
actually notices.

| Compute units | ms/image | Mean cosine | Min | Below 0.99 | Top-10 overlap |
|---|---|---|---|---|---|
| **`.cpuAndGPU`** | **224** | **0.9999** | **0.9991** | 0/134 | **98 %** |
| `.cpuOnly` | 353 | 0.9990 | 0.9817 | 2/134 | 96 % |
| `.cpuAndNeuralEngine` | 184 | 0.9946 | 0.9442 | 14/134 | 82 % |

**Use `.cpuAndGPU`.** The Neural Engine is the fastest of the three, but at
400M parameters its fp16 arithmetic drifts far enough to reorder search
results, and `.all` will pick it. The GPU is 20 % slower and reproduces the
original almost exactly.

For reference, the same ONNX model on CPU runs at 1395 ms/image, so the GPU
path is about 6× faster.

If you are converting a *smaller* SigLIP2 (the 86M-parameter base model, for
instance), this does not apply: there the Neural Engine is both the fastest and
lossless, and `.all` is the right choice. The lesson is to measure the
distribution and the ranking, not the mean cosine — the mean hid this.

## How it was converted

```bash
uv run convert_to_coreml.py ViT-SO400M-14-SigLIP2-378
```

Using the script published in
[batmac/ViT-B-16-SigLIP2-Image-CoreML](https://huggingface.co/batmac/ViT-B-16-SigLIP2-Image-CoreML):
it loads the model through `open_clip`, wraps it so the output comes out
L2-normalized, traces it with `torch.jit.trace`, and converts with
`coremltools` (`minimum_deployment_target=macOS14`).

Note that the script's own `--verify` step fails on this model: it builds a
fixed 224×224 test image, and Core ML rejects any size other than 378. Verify
against a reference implementation instead.

## License

Apache 2.0, inherited from the original model. This is a derivative work; the
only modifications are the format (PyTorch → Core ML) and the precision (fp16).