Add pre-trained Alligat0R weights (fp16 safetensors) and model card

README.md

@@ -0,0 +1,119 @@
+---
+license: mit
+pipeline_tag: image-segmentation
+tags:
+  - covisibility
+  - croco
+  - pose-estimation
+  - nuscenes
+  - scannet
+  - neurips-2025
+  - vision-transformer
+datasets:
+  - thibautloiseau/Cub3
+---
+
+# Alligat0R: Pre-Training through Covisibility Segmentation for Relative Camera Pose Regression
+
+**NeurIPS 2025 Spotlight** | [Paper](https://arxiv.org/abs/2503.07561) | [Code](https://github.com/thibautloiseau/alligat0r) | [Dataset](https://huggingface.co/datasets/thibautloiseau/Cub3)
+
+Alligat0R is a novel pre-training approach for binocular vision tasks. Instead of cross-view completion (CroCo), it uses a **covisibility segmentation** objective: for each pixel in one image, the model predicts whether the corresponding 3D point is **covisible**, **occluded**, or **outside the field of view** in the other image.
+
+## Available Variants
+
+This repository contains four pre-trained Alligat0R backbones (covisibility segmentation, before pose fine-tuning):
+
+| Subfolder | Training Data | Image Size | Description |
+|---|---|---|---|
+| `nuscenes_cub3-50` | nuScenes (Cub3-50, >= 50% overlap) | 288 x 512 | Outdoor driving |
+| `nuscenes_cub3-all` | nuScenes (Cub3-all, >= 5% overlap) | 288 x 512 | Outdoor driving, challenging pairs |
+| `scannet_cub3-50` | ScanNet (Cub3-50, >= 50% overlap) | 384 x 512 | Indoor scenes |
+| `scannet_cub3-all` | ScanNet (Cub3-all, >= 5% overlap) | 384 x 512 | Indoor scenes, challenging pairs |
+
+All models use a ViT-Large encoder (24 layers, 1024-dim) and a ViT-Base decoder (12 layers, 768-dim) with RoPE positional embeddings. Weights are stored in fp16 safetensors format (~798 MB each).
+
+## Usage
+
+```python
+import torch
+from reloc3r.alligat0r import Alligat0R
+
+device = "cuda"
+model = Alligat0R.from_pretrained(
+    "thibautloiseau/alligat0r",
+    subfolder="scannet_cub3-all",
+    device=device,
+)
+
+from PIL import Image
+from torchvision import transforms
+
+img_size = (384, 512)  # use (288, 512) for nuScenes variants
+tf = transforms.Compose([
+    transforms.Resize(img_size),
+    transforms.ToTensor(),
+    transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225]),
+])
+view1 = {"img": tf(Image.open("img1.jpg").convert("RGB")).unsqueeze(0).to(device)}
+view2 = {"img": tf(Image.open("img2.jpg").convert("RGB")).unsqueeze(0).to(device)}
+
+with torch.no_grad():
+    seg1, seg2 = model(view1, view2)  # (B, 3, H, W) logits per view
+
+cov1 = seg1.argmax(1)[0].cpu().numpy()  # 0=covisible, 1=occluded, 2=outside-FOV
+cov2 = seg2.argmax(1)[0].cpu().numpy()
+```
+
+## Fine-Tuning for Pose Regression
+
+These pre-trained weights serve as initialization for downstream tasks. To fine-tune for relative pose regression, use the training script from the [GitHub repository](https://github.com/thibautloiseau/alligat0r):
+
+```bash
+torchrun --nproc_per_node=4 finetune_pose.py \
+    --mode alligat0r_pose \
+    --dataset scannet \
+    --overlap all \
+    --load_pretrained_default
+```
+
+## Architecture
+
+- **Encoder:** ViT-Large (24 layers, 1024-dim, 16 heads, patch size 16)
+- **Decoder:** ViT-Base (12 layers, 768-dim, 12 heads) with cross-attention
+- **Segmentation head:** linear projection from decoder features to 3-class per-pixel predictions
+- **Positional encoding:** RoPE (freq=100)
+
+The architecture is symmetric: both images are processed identically without masking, unlike CroCo which uses asymmetric masking.
+
+## Training Details
+
+- **Optimizer:** AdamW (lr=1.5e-4, weight_decay=0.05, betas=(0.9, 0.95))
+- **Schedule:** cosine decay with 2 epochs warmup, 25 training epochs
+- **Batch size:** 32 per GPU
+- **Loss:** cross-entropy on the 3-class covisibility prediction
+- **Hardware:** NVIDIA A100 GPUs
+
+## Results
+
+After fine-tuning for metric relative pose regression on Cub3-all (backbone unfrozen):
+
+| Method | RUBIK 5deg/0.5m | RUBIK 5deg/2m | RUBIK 10deg/5m | ScanNet 10deg/0.25m | ScanNet 10deg/0.5m | ScanNet 10deg/1m |
+|---|---|---|---|---|---|---|
+| CroCo (Cub3-50) | 12.4 | 38.3 | 66.7 | 75.7 | 87.4 | 91.5 |
+| **Alligat0R (Cub3-all)** | **24.6** | **60.3** | **81.9** | **85.5** | **92.5** | **95.1** |
+
+## Limitations
+
+- Models are trained on driving (nuScenes) and indoor (ScanNet) domains. Generalization to other domains (e.g., aerial, underwater) has not been evaluated.
+- nuScenes covisibility annotations rely on monocular depth predictions, which may contain noise on reflective surfaces, transparent objects, or distant geometry.
+
+## Citation
+
+```bibtex
+@inproceedings{loiseau2025alligat0r,
+  title={{Alligat0R}: Pre-Training through Covisibility Segmentation for Relative Camera Pose Regression},
+  author={Loiseau, Thibaut and Bourmaud, Guillaume and Lepetit, Vincent},
+  booktitle={NeurIPS},
+  year={2025}
+}
+```

nuscenes_cub3-50/config.json

@@ -0,0 +1,17 @@
+{
+  "patch_size": 16,
+  "enc_embed_dim": 1024,
+  "enc_depth": 24,
+  "enc_num_heads": 16,
+  "dec_embed_dim": 768,
+  "dec_depth": 12,
+  "dec_num_heads": 12,
+  "mlp_ratio": 4,
+  "norm_im2_in_dec": true,
+  "pos_embed": "RoPE100",
+  "mode": "alligat0r_pretrain",
+  "img_size": [
+    288,
+    512
+  ]
+}

nuscenes_cub3-50/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b52a8ec790acea60552134e5b813913d36da7f5041241ae754de5e53152b482e
+size 835895552

nuscenes_cub3-all/config.json

@@ -0,0 +1,17 @@
+{
+  "patch_size": 16,
+  "enc_embed_dim": 1024,
+  "enc_depth": 24,
+  "enc_num_heads": 16,
+  "dec_embed_dim": 768,
+  "dec_depth": 12,
+  "dec_num_heads": 12,
+  "mlp_ratio": 4,
+  "norm_im2_in_dec": true,
+  "pos_embed": "RoPE100",
+  "mode": "alligat0r_pretrain",
+  "img_size": [
+    288,
+    512
+  ]
+}

nuscenes_cub3-all/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9415a75478a8faf948cdc066a5ff63708cd17aa4eac35e6a1aa95c5166e0472e
+size 835895552

scannet_cub3-50/config.json

@@ -0,0 +1,17 @@
+{
+  "patch_size": 16,
+  "enc_embed_dim": 1024,
+  "enc_depth": 24,
+  "enc_num_heads": 16,
+  "dec_embed_dim": 768,
+  "dec_depth": 12,
+  "dec_num_heads": 12,
+  "mlp_ratio": 4,
+  "norm_im2_in_dec": true,
+  "pos_embed": "RoPE100",
+  "mode": "alligat0r_pretrain",
+  "img_size": [
+    384,
+    512
+  ]
+}

scannet_cub3-50/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:09aa0b33cecb2b8fd018b0ce0962bd2ded11188a816f0a508f1e18baf8bb5397
+size 835895552

scannet_cub3-all/config.json

@@ -0,0 +1,17 @@
+{
+  "patch_size": 16,
+  "enc_embed_dim": 1024,
+  "enc_depth": 24,
+  "enc_num_heads": 16,
+  "dec_embed_dim": 768,
+  "dec_depth": 12,
+  "dec_num_heads": 12,
+  "mlp_ratio": 4,
+  "norm_im2_in_dec": true,
+  "pos_embed": "RoPE100",
+  "mode": "alligat0r_pretrain",
+  "img_size": [
+    384,
+    512
+  ]
+}

scannet_cub3-all/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b603d4adc2acbcb254be2948b5e067ec4c1a657741321b38344c1918b09bd420
+size 835895552