| --- |
| license: apache-2.0 |
| library_name: pytorch |
| pipeline_tag: image-segmentation |
| tags: |
| - affordance-segmentation |
| - instance-segmentation |
| - object-detection |
| - robotics |
| - manipulation |
| - rt-detr |
| - real-time |
| metrics: |
| - name: Weighted F-measure (beta^2=1) |
| type: f-measure |
| value: 0.8685 |
| --- |
| |
| # AffKernel: IIT-AFF affordance segmentation (R50vd, stride-2, full model) |
|
|
| Released weights for **AffKernel**: single-pass, NMS-free affordance |
| segmentation. RT-DETR object queries generate per-query dynamic convolution |
| kernels that decode per-object affordance masks from one shared high-resolution |
| map. |
|
|
| > **This card describes the artifacts only.** Method, results, ablations, |
| > deployment benchmarks, protocol caveats, licensing detail and citation all |
| > live in the code repository and are deliberately not repeated here: |
| > **https://github.com/anh0001/affkernel** |
| > ([reproduction guide](https://github.com/anh0001/affkernel/blob/main/docs/reproduction.md), |
| > [dataset setup](https://github.com/anh0001/affkernel/blob/main/docs/datasets.md)) |
|
|
| ## What this checkpoint is |
|
|
| IIT-AFF, 72 epochs, **seed 42**, last-epoch EMA weights, fp32. |
|
|
| | | | |
| |---|---:| |
| | `F_beta^w` (beta^2 = 1) | **0.8685** | |
| | `F_beta^w` (beta^2 = 0.3) | 0.8582 | |
|
|
| This is one seed, not the headline mean: the three-seed figure is |
| 0.8675 ± 0.0009, and seed 42 is the highest of the three (7: 0.8668, |
| 123: 0.8673). Both beta conventions are given because a `beta^2 = 1` number |
| must never be compared against a `beta^2 = 0.3` one. Read the accuracy as |
| exploratory rather than confirmatory: IIT-AFF ships no validation split, so the |
| same test set also informed the recipe. Full reasoning in the repository README. |
|
|
| ## Load it |
|
|
| ```bash |
| hf download anhrisn/affkernel-iit-aff \ |
| affkernel_iit_r50vd_stride2_deepsup_seed42.pth --local-dir weights/ |
| ``` |
|
|
| Inference, evaluation and deployment commands are in the |
| [repository README](https://github.com/anh0001/affkernel#readme). |
|
|
| ## Files |
|
|
| | File | What it is | |
| |---|---| |
| | `affkernel_iit_r50vd_stride2_deepsup_seed42.pth` | The model. fp32 EMA weights, 174 MB. | |
| | `backbone_fp16.plan` | Optional TensorRT fp16 **backbone-only** engine, 46.4 MiB. Not portable, not usable on its own. | |
|
|
| `deepsup` in the filename is the historical identifier for the auxiliary readout |
| losses used during training; it is kept so the documented commands keep working. |
|
|
| ### Checkpoint format |
|
|
| A single file holding the EMA weights only: |
|
|
| ```python |
| {"ema": {"module": <OrderedDict of 760 tensors>, "updates": 111312}} |
| ``` |
|
|
| The evaluation path reads the EMA parameters, so raw weights and optimizer state |
| are not included. **Do not repack this into a bare `{"model": ...}` dict**: under |
| an EMA-enabled config the solver would then evaluate a freshly initialised EMA |
| module and score near zero. |
|
|
| The first run fetches the ImageNet-pretrained ResNet-50vd backbone from the |
| RT-DETR release artefacts, so it needs network access. Set |
| `PResNet.pretrained: False` for an offline deployment. |
|
|
| ### `backbone_fp16.plan` |
| |
| The dynamic-kernel affordance head cannot be exported to ONNX, but the backbone |
| is plain convolutions and is the largest single term of the forward pass |
| (17.7 ms to 5.3 ms). The encoder, decoder and affordance head stay in PyTorch, |
| CUDA-graphed against the engine's output buffers. Built on Jetson AGX Orin |
| (64 GB), JetPack 6.2 (L4T R36.4.7), TensorRT 10.3.0 / CUDA 12.6; input `images` |
| `(1, 3, 640, 640)` fp16 fixed; outputs `feat0..feat3` fp16. |
| |
| > **Engine files are not portable.** A `.plan` is compiled for one specific GPU, |
| > TensorRT version and input shape. On any other device, JetPack/TensorRT version |
| > or input size it will fail to deserialize, or load and behave incorrectly. It |
| > is published here only as a convenience for setups identical to the above. |
| > **If yours differs in any respect, build your own** — one command, a few |
| > minutes, and it is the supported path. See |
| > [Deployment](https://github.com/anh0001/affkernel#deployment-on-nvidia-jetson). |
| |
| ## Intended use and limitations |
| |
| Research on affordance perception and perception-guided grasping. |
| |
| - Closed vocabulary: 10 object and 9 affordance classes, listed in |
| [`docs/datasets.md`](https://github.com/anh0001/affkernel/blob/main/docs/datasets.md). |
| It will not generalise to unseen categories. |
| - IIT-AFF is a tabletop dataset. Other viewpoints, lighting or clutter regimes |
| are untested. |
| - The residual error is dominated by **missed detections** rather than by mask |
| quality. |
| - Not validated for safety-critical deployment. A predicted grasp region is a |
| perception cue, not a guarantee of a safe grasp. |
| |
| ## License |
| |
| Weights are Apache-2.0, matching the RT-DETR components and the |
| ImageNet-pretrained backbone they were initialised from. The **IIT-AFF dataset |
| is not redistributed** here; obtain it from |
| https://sites.google.com/site/iitaffdataset/. Full attribution and the citation |
| BibTeX are in the |
| [repository](https://github.com/anh0001/affkernel/blob/main/THIRD_PARTY_LICENSES.md). |
| |
