mnm-matin commited on
Commit
cb88cda
·
verified ·
1 Parent(s): 92be197

Update README with hycoclip-vit-s

Browse files
Files changed (1) hide show
  1. README.md +99 -0
README.md ADDED
@@ -0,0 +1,99 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: onnx
3
+ pipeline_tag: feature-extraction
4
+ license: cc-by-nc-4.0
5
+ tags:
6
+ - onnx
7
+ - vision
8
+ - clip
9
+ - hyperbolic
10
+ - image-embedding
11
+ - hyperboloid
12
+ - non-euclidean
13
+ - lorentz
14
+ - meru
15
+ - hycoclip
16
+ language:
17
+ - en
18
+ ---
19
+
20
+ # Hyperbolic CLIP Models (ONNX)
21
+
22
+ This repository contains **ONNX exports** of hyperbolic vision-language models for **hyperbolic image embeddings**.
23
+
24
+ ## Available Models
25
+
26
+ | Model | Architecture | Embedding Dim | Size | Path |
27
+ |-------|--------------|---------------|------|------|
28
+ | **hycoclip-vit-s** | ViT-S/16 | 513 | ~84 MB | `hycoclip-vit-s/model.onnx` |
29
+
30
+ ## Quick Start
31
+
32
+ ```python
33
+ import onnxruntime as ort
34
+ import numpy as np
35
+ from huggingface_hub import hf_hub_download
36
+
37
+ # Download a model
38
+ onnx_path = hf_hub_download(
39
+ repo_id="mnm-matin/hyperbolic-clip",
40
+ filename="hycoclip-vit-s/model.onnx" # or other model path
41
+ )
42
+
43
+ # Load and run
44
+ session = ort.InferenceSession(onnx_path)
45
+ image = np.random.rand(1, 3, 224, 224).astype(np.float32) # Your preprocessed image
46
+ embedding, curvature = session.run(None, {"image": image})
47
+
48
+ print(f"Embedding shape: {embedding.shape}") # (1, 513) - hyperboloid format
49
+ ```
50
+
51
+ ## Model Details
52
+
53
+ All models output embeddings in **Lorentz/Hyperboloid format**:
54
+ - Output: `(t, x₁...xₙ)` where `t = √(1/c + ‖x‖²)`
55
+ - Embedding dim: 513 (1 time component + 512 spatial)
56
+ - Curvature `c` is learned and exported as secondary output
57
+
58
+ ### Converting to Poincaré Ball
59
+
60
+ ```python
61
+ t = embedding[:, 0:1] # time component
62
+ x = embedding[:, 1:] # spatial components
63
+ poincare = x / (t + 1) # stereographic projection
64
+ ```
65
+
66
+ ## Usage with HyperView
67
+
68
+ ```python
69
+ import hyperview as hv
70
+ from huggingface_hub import hf_hub_download
71
+
72
+ # Download model
73
+ model_path = hf_hub_download("mnm-matin/hyperbolic-clip", "hycoclip-vit-s/model.onnx")
74
+
75
+ # Use with HyperView
76
+ ds = hv.Dataset("my_images")
77
+ ds.add_images_dir("/path/to/images")
78
+ ds.compute_embeddings(onnx_path=model_path)
79
+ hv.show(ds)
80
+ ```
81
+
82
+ ## License
83
+
84
+ **CC-BY-NC-4.0** (Non-commercial use only)
85
+
86
+ Based on:
87
+ - [PalAvik/hycoclip](https://github.com/PalAvik/hycoclip)
88
+ - [facebookresearch/meru](https://github.com/facebookresearch/meru)
89
+
90
+ ## Citation
91
+
92
+ ```bibtex
93
+ @inproceedings{desai2023hyperbolic,
94
+ title={Hyperbolic Image-Text Representations},
95
+ author={Desai, Karan and Nickel, Maximilian and Rajpurohit, Tanmay and Johnson, Justin and Vedantam, Ramakrishna},
96
+ booktitle={ICML},
97
+ year={2023}
98
+ }
99
+ ```