thibautloiseau commited on
Commit
4eee2d5
·
verified ·
1 Parent(s): 5c23a53

Add demo reference, encoder feature extraction docs

Browse files
Files changed (1) hide show
  1. README.md +20 -3
README.md CHANGED
@@ -34,8 +34,14 @@ All models use a ViT-Large encoder (24 layers, 1024-dim) and a ViT-Base decoder
34
 
35
  ## Usage
36
 
 
 
 
 
37
  ```python
38
  import torch
 
 
39
  from reloc3r.alligat0r import Alligat0R
40
 
41
  device = "cuda"
@@ -45,9 +51,6 @@ model = Alligat0R.from_pretrained(
45
  device=device,
46
  )
47
 
48
- from PIL import Image
49
- from torchvision import transforms
50
-
51
  img_size = (384, 512) # use (288, 512) for nuScenes variants
52
  tf = transforms.Compose([
53
  transforms.Resize(img_size),
@@ -64,6 +67,20 @@ cov1 = seg1.argmax(1)[0].cpu().numpy() # 0=covisible, 1=occluded, 2=outside-FOV
64
  cov2 = seg2.argmax(1)[0].cpu().numpy()
65
  ```
66
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
67
  ## Fine-Tuning for Pose Regression
68
 
69
  These pre-trained weights serve as initialization for downstream tasks. To fine-tune for relative pose regression, use the training script from the [GitHub repository](https://github.com/thibautloiseau/alligat0r):
 
34
 
35
  ## Usage
36
 
37
+ A complete demo with visualization is provided in [`demo.py`](https://github.com/thibautloiseau/alligat0r/blob/main/demo.py) on the GitHub repository.
38
+
39
+ ### Covisibility prediction
40
+
41
  ```python
42
  import torch
43
+ from PIL import Image
44
+ from torchvision import transforms
45
  from reloc3r.alligat0r import Alligat0R
46
 
47
  device = "cuda"
 
51
  device=device,
52
  )
53
 
 
 
 
54
  img_size = (384, 512) # use (288, 512) for nuScenes variants
55
  tf = transforms.Compose([
56
  transforms.Resize(img_size),
 
67
  cov2 = seg2.argmax(1)[0].cpu().numpy()
68
  ```
69
 
70
+ ### Encoder feature extraction
71
+
72
+ The pre-trained encoder can be used as a feature backbone for downstream tasks:
73
+
74
+ ```python
75
+ feat1, feat2, pos1, pos2 = model.encode(view1, view2)
76
+ # feat1, feat2: (B, N_patches, 1024) — ViT-L encoder features
77
+ # pos1, pos2: (B, N_patches, 2) — 2-D patch positions
78
+
79
+ # To get intermediate features from all 24 encoder blocks:
80
+ feat1_all, feat2_all, pos1, pos2 = model.encode(view1, view2, return_all_blocks=True)
81
+ # feat1_all: list of 24 tensors, each (B, N_patches, 1024)
82
+ ```
83
+
84
  ## Fine-Tuning for Pose Regression
85
 
86
  These pre-trained weights serve as initialization for downstream tasks. To fine-tune for relative pose regression, use the training script from the [GitHub repository](https://github.com/thibautloiseau/alligat0r):