---
license: apache-2.0
library_name: pytorch
pipeline_tag: image-segmentation
tags:
  - affordance-segmentation
  - instance-segmentation
  - object-detection
  - robotics
  - manipulation
  - rt-detr
  - real-time
metrics:
  - name: Weighted F-measure (beta^2=1)
    type: f-measure
    value: 0.8685
---

# AffKernel: IIT-AFF affordance segmentation (R50vd, stride-2, full model)

Released weights for **AffKernel**: single-pass, NMS-free affordance
segmentation. RT-DETR object queries generate per-query dynamic convolution
kernels that decode per-object affordance masks from one shared high-resolution
map.

> **This card describes the artifacts only.** Method, results, ablations,
> deployment benchmarks, protocol caveats, licensing detail and citation all
> live in the code repository and are deliberately not repeated here:
> **https://github.com/anh0001/affkernel**
> ([reproduction guide](https://github.com/anh0001/affkernel/blob/main/docs/reproduction.md),
> [dataset setup](https://github.com/anh0001/affkernel/blob/main/docs/datasets.md))

## What this checkpoint is

IIT-AFF, 72 epochs, **seed 42**, last-epoch EMA weights, fp32.

| | |
|---|---:|
| `F_beta^w` (beta^2 = 1) | **0.8685** |
| `F_beta^w` (beta^2 = 0.3) | 0.8582 |

This is one seed, not the headline mean: the three-seed figure is
0.8675 ± 0.0009, and seed 42 is the highest of the three (7: 0.8668,
123: 0.8673). Both beta conventions are given because a `beta^2 = 1` number
must never be compared against a `beta^2 = 0.3` one. Read the accuracy as
exploratory rather than confirmatory: IIT-AFF ships no validation split, so the
same test set also informed the recipe. Full reasoning in the repository README.

## Load it

```bash
hf download anhrisn/affkernel-iit-aff \
  affkernel_iit_r50vd_stride2_deepsup_seed42.pth --local-dir weights/
```

Inference, evaluation and deployment commands are in the
[repository README](https://github.com/anh0001/affkernel#readme).

## Files

| File | What it is |
|---|---|
| `affkernel_iit_r50vd_stride2_deepsup_seed42.pth` | The model. fp32 EMA weights, 174 MB. |
| `backbone_fp16.plan` | Optional TensorRT fp16 **backbone-only** engine, 46.4 MiB. Not portable, not usable on its own. |

`deepsup` in the filename is the historical identifier for the auxiliary readout
losses used during training; it is kept so the documented commands keep working.

### Checkpoint format

A single file holding the EMA weights only:

```python
{"ema": {"module": <OrderedDict of 760 tensors>, "updates": 111312}}
```

The evaluation path reads the EMA parameters, so raw weights and optimizer state
are not included. **Do not repack this into a bare `{"model": ...}` dict**: under
an EMA-enabled config the solver would then evaluate a freshly initialised EMA
module and score near zero.

The first run fetches the ImageNet-pretrained ResNet-50vd backbone from the
RT-DETR release artefacts, so it needs network access. Set
`PResNet.pretrained: False` for an offline deployment.

### `backbone_fp16.plan`

The dynamic-kernel affordance head cannot be exported to ONNX, but the backbone
is plain convolutions and is the largest single term of the forward pass
(17.7 ms to 5.3 ms). The encoder, decoder and affordance head stay in PyTorch,
CUDA-graphed against the engine's output buffers. Built on Jetson AGX Orin
(64 GB), JetPack 6.2 (L4T R36.4.7), TensorRT 10.3.0 / CUDA 12.6; input `images`
`(1, 3, 640, 640)` fp16 fixed; outputs `feat0..feat3` fp16.

> **Engine files are not portable.** A `.plan` is compiled for one specific GPU,
> TensorRT version and input shape. On any other device, JetPack/TensorRT version
> or input size it will fail to deserialize, or load and behave incorrectly. It
> is published here only as a convenience for setups identical to the above.
> **If yours differs in any respect, build your own** — one command, a few
> minutes, and it is the supported path. See
> [Deployment](https://github.com/anh0001/affkernel#deployment-on-nvidia-jetson).

## Intended use and limitations

Research on affordance perception and perception-guided grasping.

- Closed vocabulary: 10 object and 9 affordance classes, listed in
  [`docs/datasets.md`](https://github.com/anh0001/affkernel/blob/main/docs/datasets.md).
  It will not generalise to unseen categories.
- IIT-AFF is a tabletop dataset. Other viewpoints, lighting or clutter regimes
  are untested.
- The residual error is dominated by **missed detections** rather than by mask
  quality.
- Not validated for safety-critical deployment. A predicted grasp region is a
  perception cue, not a guarantee of a safe grasp.

## License

Weights are Apache-2.0, matching the RT-DETR components and the
ImageNet-pretrained backbone they were initialised from. The **IIT-AFF dataset
is not redistributed** here; obtain it from
https://sites.google.com/site/iitaffdataset/. Full attribution and the citation
BibTeX are in the
[repository](https://github.com/anh0001/affkernel/blob/main/THIRD_PARTY_LICENSES.md).