Ficha del modelo

README.md

@@ -0,0 +1,98 @@
+---
+license: apache-2.0
+base_model: google/siglip2-so400m-patch14-384
+tags:
+  - coreml
+  - siglip2
+  - image-encoder
+  - apple-silicon
+---
+
+# SigLIP2-so400m image encoder, Core ML
+
+The image tower of SigLIP2-so400m converted to Core ML (fp16), so it can run
+with hardware acceleration on Macs. The weights are Google's, untouched — only
+the format changed.
+
+Converted from [`open_clip`](https://github.com/mlfoundations/open_clip),
+model `ViT-SO400M-14-SigLIP2-378`, pretrained tag `webli`.
+
+This is the **image tower only**. For image↔text search you also need the
+matching text tower, which is not in this repository.
+
+## Interface
+
+| | |
+|---|---|
+| Input | `image`, an image of **378 × 378** |
+| Output | `embedding`, **1152** dimensions, **float16**, already L2-normalized |
+| Size | 815 MB |
+| Minimum target | macOS 14 |
+
+Two things are easy to get wrong, and neither one fails loudly — you just get
+worse search results:
+
+**The input is 378, not 384.** The original model is named `patch14-384`, but
+its patch embedding is a 14×14 convolution with stride 14: 27 patches fit, and
+the last 6 pixels of each dimension are dropped. So the model effectively looks
+at the top-left 378×378 region of a 384-resized image, which is also what it
+saw during training. If you resize your image to 378 directly instead of
+resizing to 384 and cropping to 378, cosine similarity against the original
+model drops from 0.998 to 0.970.
+
+**The output is float16.** Reading the `MLMultiArray` buffer as float32 gives
+you a plausible-looking vector that has nothing to do with the real embedding —
+we measured 0.02 cosine similarity against the reference. Check `dataType`
+rather than assuming.
+
+Normalization is baked into the model (scale 2/255, bias −1, i.e. SigLIP2's
+mapping to [−1, 1]), so pass raw pixels; do not normalize them yourself.
+
+## Fidelity, and which compute unit to use
+
+Measured on an M2 Pro against the fp16 ONNX export of the same model
+([onnx-community/siglip2-so400m-patch14-384-ONNX](https://huggingface.co/onnx-community/siglip2-so400m-patch14-384-ONNX)),
+over 134 frames drawn from 30 different videos, with identical preprocessing on
+both sides. "Top-10 overlap" is how much the ranking of those 134 frames agrees
+with the ONNX ranking across 30 real search queries — which is what a user
+actually notices.
+
+| Compute units | ms/image | Mean cosine | Min | Below 0.99 | Top-10 overlap |
+|---|---|---|---|---|---|
+| **`.cpuAndGPU`** | **224** | **0.9999** | **0.9991** | 0/134 | **98 %** |
+| `.cpuOnly` | 353 | 0.9990 | 0.9817 | 2/134 | 96 % |
+| `.cpuAndNeuralEngine` | 184 | 0.9946 | 0.9442 | 14/134 | 82 % |
+
+**Use `.cpuAndGPU`.** The Neural Engine is the fastest of the three, but at
+400M parameters its fp16 arithmetic drifts far enough to reorder search
+results, and `.all` will pick it. The GPU is 20 % slower and reproduces the
+original almost exactly.
+
+For reference, the same ONNX model on CPU runs at 1395 ms/image, so the GPU
+path is about 6× faster.
+
+If you are converting a *smaller* SigLIP2 (the 86M-parameter base model, for
+instance), this does not apply: there the Neural Engine is both the fastest and
+lossless, and `.all` is the right choice. The lesson is to measure the
+distribution and the ranking, not the mean cosine — the mean hid this.
+
+## How it was converted
+
+```bash
+uv run convert_to_coreml.py ViT-SO400M-14-SigLIP2-378
+```
+
+Using the script published in
+[batmac/ViT-B-16-SigLIP2-Image-CoreML](https://huggingface.co/batmac/ViT-B-16-SigLIP2-Image-CoreML):
+it loads the model through `open_clip`, wraps it so the output comes out
+L2-normalized, traces it with `torch.jit.trace`, and converts with
+`coremltools` (`minimum_deployment_target=macOS14`).
+
+Note that the script's own `--verify` step fails on this model: it builds a
+fixed 224×224 test image, and Core ML rejects any size other than 378. Verify
+against a reference implementation instead.
+
+## License
+
+Apache 2.0, inherited from the original model. This is a derivative work; the
+only modifications are the format (PyTorch → Core ML) and the precision (fp16).