| # Serving RF-DETR-base on Blackhole with tt-model-manager |
|
|
| This repo is a **tt-model container package source**: `tt-model.yaml` + `code/rf_detr` |
| (the tt-nn port, with the FastAPI server under `code/rf_detr/server/`). It packages into |
| an OCI image that `tt-model serve` / `tt serve changh95/rf-detr-p150` run on a |
| Blackhole p150a host. Weights (`Roboflow/rf-detr-base`, Apache-2.0, ~129 MB) are a pinned |
| pointer, never baked into the image. |
|
|
| | | | |
| |---|---| |
| | tt-metal tree | `/home/deepgadget/experiments/gbp-tt/tt-metal` (main `8b98410e730`, v0.78.0-dev20260820-25; torch 2.11.0+cpu) | |
| | weights | `Roboflow/rf-detr-base` @ `7b95b089788e6c7db56d5ea9b0a07ca08ea6ac0a` (`model.safetensors`, `config.json`) | |
| | app | `rf_detr.server.app:app` (uvicorn, ASGI lifespan does weights -> device -> model -> warm-up) | |
| | device recipe | `ttnn.open_device(device_id, l1_small_size=32768, trace_region_size=90_000_000, num_command_queues=1)` — the port's validated params (`conftest.py`, `scripts/make_demo.py`, `benchmark.py`) | |
| | port source | github.com/changh95/tt-RF-DETR @ `5b0149869465892477141b4c1dfc4622db0d1ec5` (+ `rf_detr/server/` added here) | |
|
|
| Below, `$ROOT=/home/deepgadget/experiments/tt-models`. |
|
|
| ## 1. Run on the HOST (hardware validation, no Docker) |
|
|
| The tree venv `$TT_METAL_HOME/python_env` (py3.10, torch 2.11.0+cpu, ttnn editable) has |
| everything except `fastapi`/`uvicorn`. Do not install into the tree venv; put the HTTP |
| stack in a side directory and prepend it to `PYTHONPATH`: |
|
|
| ```bash |
| export TT_METAL_HOME=/home/deepgadget/experiments/gbp-tt/tt-metal |
| export PATH=$HOME/.local/bin:$PATH # uv |
| mkdir -p /tmp/rfdetr-http && uv pip install --python $TT_METAL_HOME/python_env/bin/python \ |
| --target /tmp/rfdetr-http fastapi uvicorn |
| |
| cd $ROOT/models/rf-detr-p150 |
| export PYTHONPATH=$PWD/code:$TT_METAL_HOME:$TT_METAL_HOME/ttnn:$TT_METAL_HOME/tools:/tmp/rfdetr-http |
| export HF_MODEL=Roboflow/rf-detr-base |
| export TT_WEIGHTS_REVISION=7b95b089788e6c7db56d5ea9b0a07ca08ea6ac0a # already in ~/.cache/huggingface |
| export TT_MESH_SHAPE=1x1 |
| export TT_DEVICE_ID=0 # optional (default 0) |
| export TT_METAL_VISIBLE_DEVICES=0 # optional, one-card box |
| |
| $TT_METAL_HOME/python_env/bin/python -m uvicorn --host 0.0.0.0 --port 20000 --lifespan on rf_detr.server.app:app |
| ``` |
|
|
| Boot log landmarks: `Loading weights` -> `Opening device` -> `TT model built` -> |
| `Warming up: capturing trace` -> `Warmup complete` -> uvicorn `Application startup complete`. |
| The first boot compiles kernels (minutes on a cold `~/.cache/ttnn` / `TT_METAL_CACHE`); |
| later boots take seconds. Startup failures raise and uvicorn exits non-zero (no CPU |
| fallback). Stop with Ctrl-C / SIGTERM: the lifespan releases the trace, drops the device |
| tensors and calls `ttnn.close_device`. |
|
|
| Then, from a second shell (stdlib only, any Python): |
|
|
| ```bash |
| python code/rf_detr/server/smoke_test.py --url http://127.0.0.1:20000 # PASS/FAIL one-liner |
| python code/rf_detr/server/smoke_test.py --url http://127.0.0.1:20000 --wait 1800 --out /tmp/rfdetr.json |
| ``` |
|
|
| Expected on the demo image (`media/demo_source.png`, 640x480): `cat` 0.96 / 0.94, |
| `remote` 0.90 / 0.73, `couch` 0.67 — `PASS n=5 counts={"cat": 2, "couch": 1, "remote": 2} ...`. |
| The CPU reference gives cat 0.960/0.933, remote 0.898/0.728, couch 0.673 for the same input. |
|
|
| Offline override (no Hub, no HF cache): `TT_RF_DETR_WEIGHTS=<dir holding model.safetensors + config.json>`. |
|
|
| Offline checks that need no device (what the implement phase ran): |
|
|
| ```bash |
| # manifest + launcher preview (from the repo dir — `root: code` is CWD-relative) |
| $ROOT/.venv/bin/python -c "from tt_kernel.container_manifest import load_container_manifest; m = load_container_manifest('tt-model.yaml', check_sources=True); p = m.resolve_profile(); print('VALID', m.name, m.kind, p.hardware, p.mesh_device, m.weights_ref); from tt_kernel.launchers import launcher_for; L = launcher_for(m.kind); print(L.install_lines(m)); print(L.verify_lines(m))" |
| # import the app with no device |
| PYTHONPATH=$PWD/code:$TT_METAL_HOME:$TT_METAL_HOME/ttnn:$TT_METAL_HOME/tools:/tmp/rfdetr-http $TT_METAL_HOME/python_env/bin/python -c "import rf_detr.server.app as a; assert a.app" |
| ``` |
|
|
| ## 2. Package, serve, push (Docker) |
|
|
| Rootless Docker on this box needs `source $ROOT/bin/docker-env.sh` first (sets PATH and |
| `DOCKER_HOST`; the bare `docker` on PATH is podman). **Run every `tt-model` command from |
| this directory**: `source.tt_metal` and `source.extra_code[].root: code` resolve against |
| the process CWD, not against the YAML. `--out` points outside the git checkout because |
| `stage()` deletes `<out>/rf-detr-p150` before rebuilding. |
|
|
| ```bash |
| source $ROOT/bin/docker-env.sh |
| cd $ROOT/models/rf-detr-p150 |
| |
| tt-model package --container tt-model.yaml --out $ROOT/build # builds the image (hours cold), runs verify.sh |
| tt-model serve $ROOT/build/rf-detr-p150/tt_kernel_manifest.json # pre-downloads the pinned weights, boots, waits for READY |
| python code/rf_detr/server/smoke_test.py --url http://127.0.0.1:<port serve printed> |
| tt-model stop changh95/rf-detr-p150 |
| tt-model push $ROOT/build/rf-detr-p150 --publish # uploads code/, image/, README (generated card) |
| ``` |
|
|
| `serve` publishes the first free port from 20000 and exports into the container exactly: |
| `HF_MODEL=Roboflow/rf-detr-base`, `MESH_DEVICE=P150`, `TT_MESH_SHAPE=1x1`, then `serve.env` |
| (`TT_WEIGHTS_REVISION=7b95b0…`, `TT_METAL_VISIBLE_DEVICES=0`). The HF cache is mounted at |
| `/hf` (rw), kernels at `/cache`, and `/weight-cache` is unused by this port (weights are |
| converted to bf16/bf8 device tensors at boot in seconds). tt-cli users: |
| `tt serve changh95/rf-detr-p150` / `tt model stop changh95/rf-detr-p150`. |
|
|
| `push` makes `code/` and `image/` on the Hub exactly the staged trees (`rf_detr`, |
| `scripts`, `conftest.py` and the `models/common/lightweightmodule.py` filler) and replaces |
| `README.md` with the generated card (everything worth keeping lives in |
| `card.description` / `card.quickstart`). `media/`, `SERVING.md`, `.gitattributes` and |
| `tt-model.yaml` at the repo root survive; the orchestrator restores the `license` / |
| `pipeline_tag` front matter afterwards. |
|
|
| ## 3. The request / response contract |
|
|
| | route | purpose | |
| |---|---| |
| | `GET /health` | `{"status": "ok" \| "starting", "model": "RF-DETR-base", "device": "blackhole p150 (device_id=0, mesh 1x1)" \| null}` — 200 always; `ok` only after warm-up | |
| | `GET /info` | model, task, hardware, weights repo + revision + resolved files, port source + commit, input limits (560x560, 1 image, batch 1), COCO labels, device params | |
| | `GET /v1/models` | `{"object": "list", "data": [{"id": "Roboflow/rf-detr-base", "object": "model", "owned_by": "changh95"}]}` — only so the OpenAI-shaped ready card / `tt-model curl` do not 404 | |
| | `POST /predict` | one image -> detections (below) | |
|
|
| `POST /predict` request (JSON): |
|
|
| | field | type | default | meaning | |
| |---|---|---|---| |
| | `image` | str | required | base64 PNG/JPEG, any size (max side 8192, max 32 MB); a `data:image/...;base64,` prefix and wrapped base64 are tolerated | |
| | `threshold` | float 0..1 | `0.5` | keep detections with `sigmoid(logit) > threshold` | |
| | `max_detections` | int 1..300 | `100` | keep at most this many (highest scores first) | |
| | `return_raw` | bool | `false` | also return `raw.logits` `[300][91]` and `raw.pred_boxes` `[300][4]` (normalised cxcywh) | |
|
|
| Response (200): |
|
|
| ```json |
| { |
| "detections": [ |
| {"label": "cat", "label_id": 17, "score": 0.96, |
| "box": [7.35, 54.64, 318.45, 472.14], "box_cxcywh_norm": [0.2545, 0.5487, 0.4861, 0.8698]} |
| ], |
| "num_detections": 5, |
| "image_size": {"width": 640, "height": 480}, |
| "input_size": [560, 560], |
| "threshold": 0.5, |
| "timing_ms": {"decode": 3.1, "preprocess": 4.0, "inference": 47.0, "postprocess": 0.4, "total": 54.5} |
| } |
| ``` |
|
|
| `box` is `[x1, y1, x2, y2]` in pixels of the ORIGINAL image (the preprocessor squashes the |
| whole image to 560x560, so the model's normalised cxcywh is relative to the original too; |
| clamped to the image bounds), sorted by score descending. `label_id` is the COCO-91 index |
| from `config.json`'s `id2label`. Errors: **400** undecodable/oversized image, **422** |
| schema violation (missing `image`, `threshold` out of range), **503** while starting, |
| **500** with `inference failed: <ExceptionType>: <text>` if the device call raises. |
|
|
| Server-side pipeline (the port's validated recipe): PIL RGB -> bilinear+antialias resize |
| to 560x560 -> /255 -> ImageNet mean/std -> `TtRfDetr(pixel_values)` under one global |
| `threading.Lock` and `torch.inference_mode()` -> `sigmoid(logits).max(-1)` -> threshold -> |
| sort -> scale cxcywh by (W, H). |
|
|
| ## 4. Caveats |
|
|
| - **Batch 1, one image per request.** The TT graph is shape-locked to 560x560 |
| (16 windows x 101 tokens, 40x40 feature grid, 300 queries, 91 classes). Requests are |
| serialised by a lock; concurrent clients queue. |
| - **Warm-up captures the metal-trace** of the projector+transformer tail inside the |
| lifespan (two forwards on a synthetic 560x560 image), so READY means warm; the first |
| cold boot pays the ttnn JIT (minutes), later boots reuse `/cache` (`TT_METAL_CACHE`). |
| - **Weights are pinned by sha** in both `weights.revision` and `serve.env.TT_WEIGHTS_REVISION`; |
| `hf_hub_download(..., revision=<sha>)` is a pure cache hit after `serve`'s pre-download |
| and falls back to `local_files_only=True` if the Hub is unreachable. |
| - **Precision knobs stay off.** `RFDETR_BB_FIDELITY` / `RFDETR_BB_FP32ACC` / `RFDETR_BB_L1` |
| are read by `TtRfDetr` but every setting other than the default regressed accuracy or hung |
| (port README); do not put them in `serve.env`. |
| - **ttnn API drift** is the main hardware-phase risk: the port was validated on a |
| ~June-2026 tree; the packaged tree is 2026-08-20. `Conv2dConfig.shard_layout`, |
| `ttnn.topk` index dtype, `execute_trace` kwargs and `grid_sample` kwargs all exist in |
| this tree (checked offline); numerics/gate (98.5 detection-IoU) must be re-confirmed |
| with `pytest code/rf_detr/tests/test_pretrained_eval.py` or the smoke test on the box. |
| - The `tt_metal/python_env` venv on this host is Python 3.10 (the image uses 3.12); the |
| code is version-agnostic (3.9+ syntax, `from __future__ import annotations`). |
| - `tt-model curl` and the ready card's `/v1/models` hint are OpenAI-shaped and are not this |
| API; use the routes above. |
|
|
