Add demo reference, encoder feature extraction docs
Browse files
README.md
CHANGED
|
@@ -34,8 +34,14 @@ All models use a ViT-Large encoder (24 layers, 1024-dim) and a ViT-Base decoder
|
|
| 34 |
|
| 35 |
## Usage
|
| 36 |
|
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
```python
|
| 38 |
import torch
|
|
|
|
|
|
|
| 39 |
from reloc3r.alligat0r import Alligat0R
|
| 40 |
|
| 41 |
device = "cuda"
|
|
@@ -45,9 +51,6 @@ model = Alligat0R.from_pretrained(
|
|
| 45 |
device=device,
|
| 46 |
)
|
| 47 |
|
| 48 |
-
from PIL import Image
|
| 49 |
-
from torchvision import transforms
|
| 50 |
-
|
| 51 |
img_size = (384, 512) # use (288, 512) for nuScenes variants
|
| 52 |
tf = transforms.Compose([
|
| 53 |
transforms.Resize(img_size),
|
|
@@ -64,6 +67,20 @@ cov1 = seg1.argmax(1)[0].cpu().numpy() # 0=covisible, 1=occluded, 2=outside-FOV
|
|
| 64 |
cov2 = seg2.argmax(1)[0].cpu().numpy()
|
| 65 |
```
|
| 66 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 67 |
## Fine-Tuning for Pose Regression
|
| 68 |
|
| 69 |
These pre-trained weights serve as initialization for downstream tasks. To fine-tune for relative pose regression, use the training script from the [GitHub repository](https://github.com/thibautloiseau/alligat0r):
|
|
|
|
| 34 |
|
| 35 |
## Usage
|
| 36 |
|
| 37 |
+
A complete demo with visualization is provided in [`demo.py`](https://github.com/thibautloiseau/alligat0r/blob/main/demo.py) on the GitHub repository.
|
| 38 |
+
|
| 39 |
+
### Covisibility prediction
|
| 40 |
+
|
| 41 |
```python
|
| 42 |
import torch
|
| 43 |
+
from PIL import Image
|
| 44 |
+
from torchvision import transforms
|
| 45 |
from reloc3r.alligat0r import Alligat0R
|
| 46 |
|
| 47 |
device = "cuda"
|
|
|
|
| 51 |
device=device,
|
| 52 |
)
|
| 53 |
|
|
|
|
|
|
|
|
|
|
| 54 |
img_size = (384, 512) # use (288, 512) for nuScenes variants
|
| 55 |
tf = transforms.Compose([
|
| 56 |
transforms.Resize(img_size),
|
|
|
|
| 67 |
cov2 = seg2.argmax(1)[0].cpu().numpy()
|
| 68 |
```
|
| 69 |
|
| 70 |
+
### Encoder feature extraction
|
| 71 |
+
|
| 72 |
+
The pre-trained encoder can be used as a feature backbone for downstream tasks:
|
| 73 |
+
|
| 74 |
+
```python
|
| 75 |
+
feat1, feat2, pos1, pos2 = model.encode(view1, view2)
|
| 76 |
+
# feat1, feat2: (B, N_patches, 1024) — ViT-L encoder features
|
| 77 |
+
# pos1, pos2: (B, N_patches, 2) — 2-D patch positions
|
| 78 |
+
|
| 79 |
+
# To get intermediate features from all 24 encoder blocks:
|
| 80 |
+
feat1_all, feat2_all, pos1, pos2 = model.encode(view1, view2, return_all_blocks=True)
|
| 81 |
+
# feat1_all: list of 24 tensors, each (B, N_patches, 1024)
|
| 82 |
+
```
|
| 83 |
+
|
| 84 |
## Fine-Tuning for Pose Regression
|
| 85 |
|
| 86 |
These pre-trained weights serve as initialization for downstream tasks. To fine-tune for relative pose regression, use the training script from the [GitHub repository](https://github.com/thibautloiseau/alligat0r):
|