anhrisn commited on
Commit
5c9fd63
·
verified ·
1 Parent(s): 07feac4

Model card: cut what the code repo already documents

Browse files

Removes the sections duplicated from the GitHub README (usage command block, class lists, beta-convention explanation, licence detail, full citation BibTeX) and links there instead. Keeps only what is specific to these artifacts: which seed and epoch, checkpoint dict layout, the non-portable TensorRT engine, and the model-card limitations. 198 -> 124 lines.

Files changed (1) hide show
  1. README.md +38 -112
README.md CHANGED
@@ -21,79 +21,47 @@ metrics:
21
  Released weights for **AffKernel**: single-pass, NMS-free affordance
22
  segmentation. RT-DETR object queries generate per-query dynamic convolution
23
  kernels that decode per-object affordance masks from one shared high-resolution
24
- map. A detector tells a robot *what* an object is; AffKernel also tells it
25
- *where* to act on it.
26
 
27
- **Documentation lives in the code repository. This card covers the artifacts
28
- only.**
29
-
30
- | | |
31
- |---|---|
32
- | Code, results, deployment guide | https://github.com/anh0001/affkernel |
33
- | Reproduction guide | [`docs/reproduction.md`](https://github.com/anh0001/affkernel/blob/main/docs/reproduction.md) |
34
- | Dataset setup | [`docs/datasets.md`](https://github.com/anh0001/affkernel/blob/main/docs/datasets.md) |
35
 
36
  ## What this checkpoint is
37
 
38
- Trained on IIT-AFF (6,184 train images), 72 epochs, seed 42, last-epoch EMA
39
- weights, fp32.
40
 
41
- | Metric (IIT-AFF test, 2,651 images) | Value |
42
  |---|---:|
43
- | `F_beta^w` (beta^2 = 1), this checkpoint | **0.8685** |
44
- | `F_beta^w` (beta^2 = 0.3), this checkpoint | 0.8582 |
45
- | Model-only latency, 640x640, fp32, RTX 6000 Ada | 16.3 ms |
46
-
47
- For context the three-seed mean is 0.8675 ± 0.0009; this is seed 42, the paper's
48
- anchor seed and the highest of the three (seed 7: 0.8668, seed 123: 0.8673). The
49
- epoch was fixed a priori (last epoch, EMA) rather than selected on the test set.
50
 
51
- Two `beta` conventions circulate in this literature, so both are given. Never
52
- compare a `beta^2 = 1` number against a `beta^2 = 0.3` one.
 
 
 
 
53
 
54
- > **These results are exploratory, not confirmatory.** IIT-AFF ships no
55
- > validation split, so the same test set that produced this number also informed
56
- > the recipe as it evolved and the choice of reported configuration. Fixing the
57
- > last epoch a priori prevents peak-checkpoint selection but not that broader
58
- > adaptive use. Read it as evidence from a single benchmark, not as an estimate
59
- > of generalisation.
60
-
61
- Baseline comparisons, the readout-resolution ladder, the full deployment
62
- benchmarks and the ablations are all in the
63
- [repository README](https://github.com/anh0001/affkernel#readme). Peer latencies
64
- there are measured on each method's own hardware and are not normalised, so no
65
- speed ratio between them is meaningful.
66
-
67
- ## Usage
68
 
69
  ```bash
70
- git clone https://github.com/anh0001/affkernel.git
71
- cd affkernel
72
- pip install -r requirements.txt huggingface_hub
73
-
74
  hf download anhrisn/affkernel-iit-aff \
75
  affkernel_iit_r50vd_stride2_deepsup_seed42.pth --local-dir weights/
76
-
77
- python tools/infer.py \
78
- -c configs/rtdetr/rtdetr_r50vd_6x_iit_v3_stride2_deepsup.yml \
79
- -r weights/affkernel_iit_r50vd_stride2_deepsup_seed42.pth \
80
- --input path/to/image.jpg \
81
- --output outputs/
82
  ```
83
 
84
- `--input` takes an image or a directory; `--output` is a directory, written as
85
- `<stem>_aff.png`. Add `--device cpu` to run without a GPU.
86
-
87
- The first run downloads the ImageNet-pretrained ResNet-50vd backbone from the
88
- RT-DETR release artefacts, so it needs network access. Set
89
- `PResNet.pretrained: False` for an offline deployment.
90
 
91
  ## Files
92
 
93
  | File | What it is |
94
  |---|---|
95
  | `affkernel_iit_r50vd_stride2_deepsup_seed42.pth` | The model. fp32 EMA weights, 174 MB. |
96
- | `backbone_fp16.plan` | Optional. TensorRT fp16 **backbone-only** engine, 46.4 MiB. Not portable, not usable on its own. |
97
 
98
  `deepsup` in the filename is the historical identifier for the auxiliary readout
99
  losses used during training; it is kept so the documented commands keep working.
@@ -111,6 +79,10 @@ are not included. **Do not repack this into a bare `{"model": ...}` dict**: unde
111
  an EMA-enabled config the solver would then evaluate a freshly initialised EMA
112
  module and score near zero.
113
 
 
 
 
 
114
  ### `backbone_fp16.plan`
115
 
116
  The dynamic-kernel affordance head cannot be exported to ONNX, but the backbone
@@ -128,71 +100,25 @@ CUDA-graphed against the engine's output buffers. Built on Jetson AGX Orin
128
  > minutes, and it is the supported path. See
129
  > [Deployment](https://github.com/anh0001/affkernel#deployment-on-nvidia-jetson).
130
 
131
- ## Classes
132
-
133
- **Objects (10):** bowl, tvm, pan, hammer, knife, cup, drill, racket, spatula, bottle
134
-
135
- **Affordances (9):** contain, cut, display, engine, grasp, hit, pound, support, w-grasp
136
-
137
  ## Intended use and limitations
138
 
139
- Intended for research on affordance perception and perception-guided grasping.
140
 
141
- - Closed vocabulary of 10 object and 9 affordance classes; it will not
142
- generalise to unseen categories.
 
143
  - IIT-AFF is a tabletop dataset. Other viewpoints, lighting or clutter regimes
144
  are untested.
145
  - The residual error is dominated by **missed detections** rather than by mask
146
- quality. On instances the detector does fire on, mask quality reaches 0.893;
147
- that figure excludes detection misses, so it is not protocol-matched to
148
- published full-set numbers and is not a state-of-the-art claim.
149
- - The headline latency is fp32 on an RTX 6000 Ada. fp16, CUDA graphs and the
150
- TensorRT backbone are characterised on that machine and on a Jetson AGX Orin
151
- (23.1 FPS end to end within 0.23 GiB) and cost at most 0.0007 `F_beta^w`,
152
- below the seed-to-seed spread.
153
  - Not validated for safety-critical deployment. A predicted grasp region is a
154
  perception cue, not a guarantee of a safe grasp.
155
 
156
- ## License and attribution
157
-
158
- These weights are Apache-2.0, matching the RT-DETR components and the
159
- ImageNet-pretrained backbone they were initialised from. The repository's own
160
- source contributions are MIT; see
161
- [`THIRD_PARTY_LICENSES.md`](https://github.com/anh0001/affkernel/blob/main/THIRD_PARTY_LICENSES.md).
162
-
163
- The **IIT-AFF dataset is not redistributed** here or in the code repository. It
164
- states no licence and its authors request citation of the original paper. Obtain
165
- it from https://sites.google.com/site/iitaffdataset/.
166
-
167
- ## Citation
168
-
169
- ```bibtex
170
- @article{risnumawan2026affkernel,
171
- title = {AffKernel: High-Resolution Readout for Real-Time Visual
172
- Affordance Segmentation},
173
- author = {Risnumawan, Anhar and Aji, Achmad Fahrul and
174
- Fatahillah, Teuku Zikri and Kubota, Naoyuki},
175
- journal = {Expert Systems with Applications},
176
- year = {2026},
177
- note = {Under review}
178
- }
179
-
180
- @inproceedings{nguyen2017object,
181
- title = {Object-Based Affordances Detection with Convolutional Neural
182
- Networks and Dense Conditional Random Fields},
183
- author = {Nguyen, Anh and Kanoulas, Dimitrios and Caldwell, Darwin G. and
184
- Tsagarakis, Nikos G.},
185
- booktitle = {IEEE/RSJ International Conference on Intelligent Robots and
186
- Systems (IROS)},
187
- year = {2017}
188
- }
189
-
190
- @inproceedings{lv2024detrs,
191
- title = {DETRs Beat YOLOs on Real-time Object Detection},
192
- author = {Lv, Wenyu and Zhao, Yian and Xu, Shangliang and Wei, Jinman and
193
- Wang, Guanzhong and Cui, Cheng and Du, Yuning and Dang, Qingqing
194
- and Liu, Yi},
195
- booktitle = {CVPR},
196
- year = {2024}
197
- }
198
- ```
 
21
  Released weights for **AffKernel**: single-pass, NMS-free affordance
22
  segmentation. RT-DETR object queries generate per-query dynamic convolution
23
  kernels that decode per-object affordance masks from one shared high-resolution
24
+ map.
 
25
 
26
+ > **This card describes the artifacts only.** Method, results, ablations,
27
+ > deployment benchmarks, protocol caveats, licensing detail and citation all
28
+ > live in the code repository and are deliberately not repeated here:
29
+ > **https://github.com/anh0001/affkernel**
30
+ > ([reproduction guide](https://github.com/anh0001/affkernel/blob/main/docs/reproduction.md),
31
+ > [dataset setup](https://github.com/anh0001/affkernel/blob/main/docs/datasets.md))
 
 
32
 
33
  ## What this checkpoint is
34
 
35
+ IIT-AFF, 72 epochs, **seed 42**, last-epoch EMA weights, fp32.
 
36
 
37
+ | | |
38
  |---|---:|
39
+ | `F_beta^w` (beta^2 = 1) | **0.8685** |
40
+ | `F_beta^w` (beta^2 = 0.3) | 0.8582 |
 
 
 
 
 
41
 
42
+ This is one seed, not the headline mean: the three-seed figure is
43
+ 0.8675 ± 0.0009, and seed 42 is the highest of the three (7: 0.8668,
44
+ 123: 0.8673). Both beta conventions are given because a `beta^2 = 1` number
45
+ must never be compared against a `beta^2 = 0.3` one. Read the accuracy as
46
+ exploratory rather than confirmatory: IIT-AFF ships no validation split, so the
47
+ same test set also informed the recipe. Full reasoning in the repository README.
48
 
49
+ ## Load it
 
 
 
 
 
 
 
 
 
 
 
 
 
50
 
51
  ```bash
 
 
 
 
52
  hf download anhrisn/affkernel-iit-aff \
53
  affkernel_iit_r50vd_stride2_deepsup_seed42.pth --local-dir weights/
 
 
 
 
 
 
54
  ```
55
 
56
+ Inference, evaluation and deployment commands are in the
57
+ [repository README](https://github.com/anh0001/affkernel#readme).
 
 
 
 
58
 
59
  ## Files
60
 
61
  | File | What it is |
62
  |---|---|
63
  | `affkernel_iit_r50vd_stride2_deepsup_seed42.pth` | The model. fp32 EMA weights, 174 MB. |
64
+ | `backbone_fp16.plan` | Optional TensorRT fp16 **backbone-only** engine, 46.4 MiB. Not portable, not usable on its own. |
65
 
66
  `deepsup` in the filename is the historical identifier for the auxiliary readout
67
  losses used during training; it is kept so the documented commands keep working.
 
79
  an EMA-enabled config the solver would then evaluate a freshly initialised EMA
80
  module and score near zero.
81
 
82
+ The first run fetches the ImageNet-pretrained ResNet-50vd backbone from the
83
+ RT-DETR release artefacts, so it needs network access. Set
84
+ `PResNet.pretrained: False` for an offline deployment.
85
+
86
  ### `backbone_fp16.plan`
87
 
88
  The dynamic-kernel affordance head cannot be exported to ONNX, but the backbone
 
100
  > minutes, and it is the supported path. See
101
  > [Deployment](https://github.com/anh0001/affkernel#deployment-on-nvidia-jetson).
102
 
 
 
 
 
 
 
103
  ## Intended use and limitations
104
 
105
+ Research on affordance perception and perception-guided grasping.
106
 
107
+ - Closed vocabulary: 10 object and 9 affordance classes, listed in
108
+ [`docs/datasets.md`](https://github.com/anh0001/affkernel/blob/main/docs/datasets.md).
109
+ It will not generalise to unseen categories.
110
  - IIT-AFF is a tabletop dataset. Other viewpoints, lighting or clutter regimes
111
  are untested.
112
  - The residual error is dominated by **missed detections** rather than by mask
113
+ quality.
 
 
 
 
 
 
114
  - Not validated for safety-critical deployment. A predicted grasp region is a
115
  perception cue, not a guarantee of a safe grasp.
116
 
117
+ ## License
118
+
119
+ Weights are Apache-2.0, matching the RT-DETR components and the
120
+ ImageNet-pretrained backbone they were initialised from. The **IIT-AFF dataset
121
+ is not redistributed** here; obtain it from
122
+ https://sites.google.com/site/iitaffdataset/. Full attribution and the citation
123
+ BibTeX are in the
124
+ [repository](https://github.com/anh0001/affkernel/blob/main/THIRD_PARTY_LICENSES.md).