--- license: mit datasets: - SPRIGHT-T2I/spright_coco base_model: - BeichenZhang/LongCLIP-L --- # Long-CLIP Needs Registers. And Gated MLPs. And +20M params. ### Fixing Long-CLIP's modality gap via happy little accidents. Love ❤ī¸ this CLIP? ᐅ [Buy me a coffee](https://ko-fi.com/zer0int) on Ko-Fi ☕
Or click here for address to send đŸĒ™â‚ŋ BTC ``` 3PscBrWYvrutXedLmvpcnQbE12Py8qLqMK ```
------- # ℹī¸ Update 02/June/2025: - You can now load the model with HF 'transformers'. ✅ - Unfortunately, AutoModel produced nonsense / I couldn't get "trust_remote_code=True" to work properly - (using that was suggested in response to my [pull request](https://github.com/huggingface/transformers/pull/38338) on GitHub). ## 💡 Alas, you will need to: - Download the '[hfmodel](https://huggingface.co/zer0int/LongCLIP-Registers-Gated_MLP-ViT-L-14/tree/main/hfmodel)' folder - Use it for manually importing the correct (my custom) CLIPModel code required as per the config.json - Minimal example code: ``` import torch from PIL import Image, ImageDraw import transformers from hfmodel.modeling_clip import CLIPModel from transformers import CLIPProcessor from torchvision.transforms import ToTensor import torch.nn.functional as F model = CLIPModel.from_pretrained("zer0int/LongCLIP-Registers-Gated_MLP-ViT-L-14") processor = CLIPProcessor.from_pretrained("zer0int/LongCLIP-Registers-Gated_MLP-ViT-L-14") device = torch.device("cuda" if torch.cuda.is_available() else "cpu") model = model.to(device) size = 224 im = Image.new("RGB", (size, size), (255, 255, 255)) draw = ImageDraw.Draw(im) # --------- GPT-4.1's idea of a pineapple. Need an input image... --------- body_bbox = [size*0.28, size*0.38, size*0.72, size*0.90] draw.ellipse(body_bbox, fill=(254, 221, 72), outline=(180, 120, 0), width=5) eye_color = (198, 134, 66) for row in range(4): for col in range(3): ex = size*0.36 + col*size*0.09 + (row%2)*size*0.045 ey = size*0.50 + row*size*0.085 ew, eh = size*0.035, size*0.025 draw.ellipse([ex-ew, ey-eh, ex+ew, ey+eh], fill=eye_color, outline=None) leaf_color = (61, 179, 70) leaf_base_x = size/2 leaf_base_y = size*0.38 for i, (angle, length) in enumerate([(-28, 65), (-12, 70), (0, 80), (12, 70), (28, 65)]): from math import radians, cos, sin a = radians(angle) tip_x = leaf_base_x + length*sin(a) tip_y = leaf_base_y - length*cos(a) left = (leaf_base_x + 13*cos(a+1.5), leaf_base_y + 13*sin(a+1.5)) right = (leaf_base_x + 13*cos(a-1.5), leaf_base_y + 13*sin(a-1.5)) draw.polygon([left, (tip_x, tip_y), right], fill=leaf_color) im.save("pineapple.png") # --------- image = Image.open("pineapple.png").convert("RGB") texts = ["pine", "apple", "pineapple", "orange", "pear", "person", "cat", "dog"] inputs = processor(text=texts, images=image, return_tensors="pt", padding=True).to(device) with torch.no_grad(): outputs = model(**inputs) image_embeds = outputs.image_embeds text_embeds = outputs.text_embeds image_embeds = F.normalize(image_embeds, dim=-1) text_embeds = F.normalize(text_embeds, dim=-1) cos_sim = image_embeds @ text_embeds.T cos_sim = cos_sim.squeeze(0) for text, sim in zip(texts, cos_sim): print(f"Similarity with '{text}': {sim.item():.4f}") ``` -------- ![image/png](https://cdn-uploads.huggingface.co/production/uploads/6490359a877fc29cb1b09451/yz_2pclSVpw1KGcvudGaU.png) ## I just want a new Text Encoder.. - ...for my Text-to-Image (Text-to-Video) AI! \o/ - Here you go: 👉 [direct download](https://huggingface.co/zer0int/LongCLIP-Registers-Gated_MLP-ViT-L-14/resolve/main/Long-ViT-L-14-REG-TE-only-HF-format.safetensors?download=true) 👈 - The model has 248 tokens max input (instead of CLIP-L: 77) - Replace your CLIP-L with this Long-CLIP (e.g. ComfyUI natively supports Long-CLIP). - Enjoy! (You don't need to do anything else, they're just normal CLIP Text Encoders!) ## ⚠ī¸ Full model (currently) not HuggingFace Transformers compatible. ⚠ī¸ - The ViT (Vision Encoder) is basically a big mutant. Alas: - The full model .safetensors have the 'import clip' (OpenAI) structure inside. - Alas, currently it runs with 'import clip' code (I'm working on a HF implementation, though!). - For more info, see also (not-a-long CLIP-L version, 77 tokens): [zer0int/CLIP-Registers-Gated_MLP-ViT-L-14](https://huggingface.co/zer0int/CLIP-Registers-Gated_MLP-ViT-L-14) ## ✅ Info / Using the full model - Models available: FULL, TE-only (Text Encoder only), LongCLIP-L.safetensors - LongCLIP-L the original model from [BeichenZhang/LongCLIP-L](https://huggingface.co/BeichenZhang/LongCLIP-L/tree/main). - It's just so you don't have to download a danger-pickle. :) - git clone my repo, [github.com/zer0int/CLIP-fine-tune-registers-gated](https://github.com/zer0int/CLIP-fine-tune-registers-gated) - Put the FULL model and the LongCLIP-L from this HF in a 'models' subfolder - You're all set! I made an entire playground for the models (+ safetensors loading)! 🎉 - PS: All code for fine-tuning it yourself is also included on my Git! 🤗 | Metric | LongCLIP-L Original | Long-ViT-L/14 Register-Tokens, X-GATED | |-------------------------------------|---------------------|----------------------------------------| | VoC-2007 multilabel, mAP | 0.8221 | 0.8403 | | MSCOCO Image Retrieval Recall@5 | 0.2761 | 0.3663 | | MSCOCO Text Retrieval Recall@5 | 0.3314 | 0.5398 | | CIFAR10 Linear Probe Acc@1 | 0.9809 | 0.9812 | | CIFAR10 Linear Probe Acc@5 | 0.9998 | 0.9997 | | CIFAR10 LP Mean Recall | 0.9809 | 0.9812 | | ImageNet/ObjectNet MVT (Zero-Shot) | 0.8103 | 0.8724 | | ILSVRC2012 LP, Top-1 | 66.95% | 66.84% | | ILSVRC2012 LP, Top-5 | 91.87% | 91.70% | | Modality Gap (Euclidean) | 1.0672 ⚠ī¸ | 0.5781 ✅ | | Img-Text Cosine (Mean) | 0.2666 | 0.4711 | | Img-Text Cosine (Std Dev) | 0.0191 | 0.0726 | | Txt-Text Cosine (Mean) | 0.8421 | 0.7046 | | Txt-Text Cosine (Std Dev) | 0.0707 | 0.1498 | | Jensen-Shannon Divergence (JSD) | 0.3847 | 0.1894 | | Wasserstein Distance | 0.5755 | 0.2335 |