--- license: apache-2.0 library_name: pytorch pipeline_tag: image-segmentation tags: - affordance-segmentation - instance-segmentation - object-detection - robotics - manipulation - rt-detr - real-time metrics: - name: Weighted F-measure (beta^2=1) type: f-measure value: 0.8685 --- # AffKernel: IIT-AFF affordance segmentation (R50vd, stride-2 + deep supervision) Single-pass, NMS-free affordance segmentation. AffKernel couples RT-DETR object queries with a CondInst-style per-query dynamic-convolution kernel that decodes per-object affordance masks from one shared high-resolution affordance map. A detector tells a robot *what* an object is; AffKernel also tells it *where* to act on it, at a latency a control loop can afford. - **Code:** https://github.com/anh0001/affkernel - **Architecture:** RT-DETR-R50vd + affordance branch with stride-2 readout and deep supervision - **Training data:** IIT-AFF (Nguyen et al., IROS 2017), 6,184 train images - **Weights:** last-epoch (epoch 72) EMA parameters, fp32, 174 MB ## Results Evaluated on the IIT-AFF test split (2,651 images) with the Margolin weighted F-measure. | Metric | Value | |---|---:| | `F_beta^w` (beta^2 = 1), this checkpoint | **0.8685** | | `F_beta^w` (beta^2 = 0.3), this checkpoint | 0.8582 | | `F_beta^w` (beta^2 = 1), mean of 3 seeds | 0.8675 ± 0.0009 | | Mask quality on detected instances | 0.8933 | | Latency, 640x640, fp32, RTX 6000 Ada | 23.8 ms median | | Throughput | 41.7 img/s (42 FPS) | Against published baselines on the same benchmark: Mask R-CNN 0.844 at 45 ms, deterministic Swin-T 0.883 at 42 ms, Bayesian Swin-T deep ensemble 0.906 at roughly 1015 ms. AffKernel beats the Mask R-CNN baseline at about half its latency and trails the deterministic Swin-T by 1.55 points at 57% of its latency, with fully deterministic single-pass inference. **On seed selection.** This is seed 42, which is both the primary anchor seed used throughout the paper and the highest scoring of the three seeds trained (seed 7: 0.8668, seed 123: 0.8673). IIT-AFF provides no validation split, so the epoch was fixed a priori (last epoch, EMA weights) rather than selected on the test set. **On the beta convention.** Two conventions circulate in this literature. The AffordanceNet lineage reports `beta^2 = 0.3`; recent transformer baselines report `beta^2 = 1`. Both are given above so that comparisons can be made at a matched convention. Do not compare a `beta^2 = 1` number against a `beta^2 = 0.3` number. ## Usage ```bash git clone https://github.com/anh0001/affkernel.git cd affkernel pip install -r requirements.txt pip install huggingface_hub hf download anhrisn/affkernel-iit-aff \ affkernel_iit_r50vd_stride2_deepsup_seed42.pth --local-dir weights/ python tools/infer.py \ -c configs/rtdetr/rtdetr_r50vd_6x_iit_v3_stride2_deepsup.yml \ -r weights/affkernel_iit_r50vd_stride2_deepsup_seed42.pth \ --input path/to/image.jpg \ --output outputs/prediction.png ``` Reproduce the reported metric (requires the IIT-AFF dataset; see [`docs/datasets.md`](https://github.com/anh0001/affkernel/blob/main/docs/datasets.md)): ```bash python tools/decompose_fbw_gap.py \ -c configs/rtdetr/rtdetr_r50vd_6x_iit_v3_stride2_deepsup.yml \ -r weights/affkernel_iit_r50vd_stride2_deepsup_seed42.pth --beta2 1.0 ``` ### Checkpoint format A single file containing the EMA weights only: ```python {"ema": {"module": , "updates": 111312}} ``` The evaluation path uses the EMA parameters, so the raw (non-EMA) weights and the optimizer state from training are not included. Do not repack this into a bare `{"model": ...}` dict: under an EMA-enabled config the solver would then evaluate a freshly initialised EMA module and score near zero. The first run downloads ImageNet-pretrained ResNet-50vd backbone weights from the RT-DETR release artefacts, so it needs network access. For an offline deployment, set `PResNet.pretrained: False` in the config. ## Classes **Objects (10):** bowl, tvm, pan, hammer, knife, cup, drill, racket, spatula, bottle **Affordances (9):** contain, cut, display, engine, grasp, hit, pound, support, w-grasp ## Intended use and limitations Intended for research on affordance perception and perception-guided grasping. - Trained on a closed vocabulary of 10 object and 9 affordance classes; it will not generalise to unseen categories. - IIT-AFF is a tabletop dataset. Performance under other viewpoints, lighting or clutter regimes is untested. - The residual error is dominated by **missed detections**, not by mask quality. On instances the detector does fire on, mask quality (0.893) already exceeds the deterministic Swin-T baseline's overall score. - Latency was measured at fp32 on an RTX 6000 Ada. fp16 and TensorRT are untested; they would be expected to help but are not characterised here. - Not validated for safety-critical deployment. A predicted grasp region is a perception cue, not a guarantee of a safe grasp. ## License and attribution These weights are released under Apache-2.0, matching the licence of the RT-DETR components and the ImageNet-pretrained backbone they were initialised from. The AffKernel repository's own source contributions are MIT licensed; see [`THIRD_PARTY_LICENSES.md`](https://github.com/anh0001/affkernel/blob/main/THIRD_PARTY_LICENSES.md). The **IIT-AFF dataset is not redistributed** here or in the code repository. It states no licence; its authors request citation of the original paper. Obtain it from https://sites.google.com/site/iitaffdataset/. ## Citation ```bibtex @article{risnumawan2026affkernel, title = {AffKernel: Single-Pass Affordance Segmentation with Per-Query Dynamic Convolution for Real-Time Robotic Manipulation}, author = {Risnumawan, Anhar}, journal = {IEEE Access}, year = {2026}, note = {Under review} } @inproceedings{nguyen2017object, title = {Object-Based Affordances Detection with Convolutional Neural Networks and Dense Conditional Random Fields}, author = {Nguyen, Anh and Kanoulas, Dimitrios and Caldwell, Darwin G. and Tsagarakis, Nikos G.}, booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)}, year = {2017} } @inproceedings{lv2024detrs, title = {DETRs Beat YOLOs on Real-time Object Detection}, author = {Lv, Wenyu and Zhao, Yian and Xu, Shangliang and Wei, Jinman and Wang, Guanzhong and Cui, Cheng and Du, Yuning and Dang, Qingqing and Liu, Yi}, booktitle = {CVPR}, year = {2024} } ```