tt-model authoring files

SERVING.md

@@ -1,112 +1,172 @@
-# Serving this port with tt-model-manager
+# Serving RF-DETR-base on Blackhole with tt-model-manager
 
-This repo carries everything needed to build a **tt-model container bundle** from it, so
-`tt-model pull` / `tt-model serve` and `tt-cli` can run it. Do this on a
-**Blackhole host** (amd64 Linux + Docker >= 25).
+This repo is a **tt-model container package source**: `tt-model.yaml` + `code/rf_detr`
+(the tt-nn port, with the FastAPI server under `code/rf_detr/server/`). It packages into
+an OCI image that `tt-model serve` / `tt serve changh95/rf-detr-blackhole` run on a
+Blackhole p150a host. Weights (`Roboflow/rf-detr-base`, Apache-2.0, ~129 MB) are a pinned
+pointer, never baked into the image.
 
-## Why not on a Mac
+| | |
+|---|---|
+| tt-metal tree | `/home/deepgadget/experiments/gbp-tt/tt-metal` (main `8b98410e730`, v0.78.0-dev20260820-25; torch 2.11.0+cpu) |
+| weights | `Roboflow/rf-detr-base` @ `7b95b089788e6c7db56d5ea9b0a07ca08ea6ac0a` (`model.safetensors`, `config.json`) |
+| app | `rf_detr.server.app:app` (uvicorn, ASGI lifespan does weights -> device -> model -> warm-up) |
+| device recipe | `ttnn.open_device(device_id, l1_small_size=32768, trace_region_size=90_000_000, num_command_queues=1)` — the port's validated params (`conftest.py`, `scripts/make_demo.py`, `benchmark.py`) |
+| port source | github.com/changh95/tt-RF-DETR @ `5b0149869465892477141b4c1dfc4622db0d1ec5` (+ `rf_detr/server/` added here) |
 
-`tt-model package --container` builds from `ghcr.io/.../ubuntu-22.04-dev-amd64` and a cold
-build is 2.5-4 hours. Packaging and pushing need **no card** (`require_host(need_devices=False)`),
-but they do need amd64. `tt-model serve` needs `/dev/tenstorrent/*`.
+Below, `$ROOT=/home/deepgadget/experiments/tt-models`.
 
-## Steps
+## 1. Run on the HOST (hardware validation, no Docker)
+
+The tree venv `$TT_METAL_HOME/python_env` (py3.10, torch 2.11.0+cpu, ttnn editable) has
+everything except `fastapi`/`uvicorn`. Do not install into the tree venv; put the HTTP
+stack in a side directory and prepend it to `PYTHONPATH`:
 
 ```bash
-# 0. get tt-model (not on PyPI)
-git clone https://github.com/tenstorrent/tt-model-manager && cd tt-model-manager
-python -m venv .venv && source .venv/bin/activate && pip install -e .
+export TT_METAL_HOME=/home/deepgadget/experiments/gbp-tt/tt-metal
+export PATH=$HOME/.local/bin:$PATH                 # uv
+mkdir -p /tmp/rfdetr-http && uv pip install --python $TT_METAL_HOME/python_env/bin/python \
+    --target /tmp/rfdetr-http fastapi uvicorn
+
+cd $ROOT/models/rf-detr-blackhole
+export PYTHONPATH=$PWD/code:$TT_METAL_HOME:$TT_METAL_HOME/ttnn:$TT_METAL_HOME/tools:/tmp/rfdetr-http
+export HF_MODEL=Roboflow/rf-detr-base
+export TT_WEIGHTS_REVISION=7b95b089788e6c7db56d5ea9b0a07ca08ea6ac0a   # already in ~/.cache/huggingface
+export TT_MESH_SHAPE=1x1
+export TT_DEVICE_ID=0                                                  # optional (default 0)
+export TT_METAL_VISIBLE_DEVICES=0                                      # optional, one-card box
+
+$TT_METAL_HOME/python_env/bin/python -m uvicorn --host 0.0.0.0 --port 20000 --lifespan on rf_detr.server.app:app
+```
+
+Boot log landmarks: `Loading weights` -> `Opening device` -> `TT model built` ->
+`Warming up: capturing trace` -> `Warmup complete` -> uvicorn `Application startup complete`.
+The first boot compiles kernels (minutes on a cold `~/.cache/ttnn` / `TT_METAL_CACHE`);
+later boots take seconds. Startup failures raise and uvicorn exits non-zero (no CPU
+fallback). Stop with Ctrl-C / SIGTERM: the lifespan releases the trace, drops the device
+tensors and calls `ttnn.close_device`.
+
+Then, from a second shell (stdlib only, any Python):
+
+```bash
+python code/rf_detr/server/smoke_test.py --url http://127.0.0.1:20000        # PASS/FAIL one-liner
+python code/rf_detr/server/smoke_test.py --url http://127.0.0.1:20000 --wait 1800 --out /tmp/rfdetr.json
+```
 
-# 1. pull THIS repo
-hf download changh95/rf-detr-blackhole --local-dir rf-detr-blackhole && cd rf-detr-blackhole
+Expected on the demo image (`media/demo_source.png`, 640x480): `cat` 0.96 / 0.94,
+`remote` 0.90 / 0.73, `couch` 0.67 — `PASS n=5 counts={"cat": 2, "couch": 1, "remote": 2} ...`.
+The CPU reference gives cat 0.960/0.933, remote 0.898/0.728, couch 0.673 for the same input.
 
-# 2. point the manifest at your tt-metal checkout
-$EDITOR tt-model.yaml          # source.tt_metal: /path/to/tt-metal
+Offline override (no Hub, no HF cache): `TT_RF_DETR_WEIGHTS=<dir holding model.safetensors + config.json>`.
 
-# 3. finish the serving adapter
-$EDITOR code/rf_detr/server/app.py   # see "VERIFY ON HARDWARE" markers
+Offline checks that need no device (what the implement phase ran):
 
-# 4. validate with no hardware and no build
-python -c "from tt_kernel.container_manifest import ContainerManifest as M; M.load('tt-model.yaml')"
+```bash
+# manifest + launcher preview (from the repo dir — `root: code` is CWD-relative)
+$ROOT/.venv/bin/python -c "from tt_kernel.container_manifest import load_container_manifest; m = load_container_manifest('tt-model.yaml', check_sources=True); p = m.resolve_profile(); print('VALID', m.name, m.kind, p.hardware, p.mesh_device, m.weights_ref); from tt_kernel.launchers import launcher_for; L = launcher_for(m.kind); print(L.install_lines(m)); print(L.verify_lines(m))"
+# import the app with no device
+PYTHONPATH=$PWD/code:$TT_METAL_HOME:$TT_METAL_HOME/ttnn:$TT_METAL_HOME/tools:/tmp/rfdetr-http $TT_METAL_HOME/python_env/bin/python -c "import rf_detr.server.app as a; assert a.app"
+```
+
+## 2. Package, serve, push (Docker)
+
+Rootless Docker on this box needs `source $ROOT/bin/docker-env.sh` first (sets PATH and
+`DOCKER_HOST`; the bare `docker` on PATH is podman). **Run every `tt-model` command from
+this directory**: `source.tt_metal` and `source.extra_code[].root: code` resolve against
+the process CWD, not against the YAML. `--out` points outside the git checkout because
+`stage()` deletes `<out>/rf-detr-blackhole` before rebuilding.
+
+```bash
+source $ROOT/bin/docker-env.sh
+cd $ROOT/models/rf-detr-blackhole
+
+tt-model package --container tt-model.yaml --out $ROOT/build       # builds the image (hours cold), runs verify.sh
+tt-model serve   $ROOT/build/rf-detr-blackhole/tt_kernel_manifest.json   # pre-downloads the pinned weights, boots, waits for READY
+python code/rf_detr/server/smoke_test.py --url http://127.0.0.1:<port serve printed>
+tt-model stop    changh95/rf-detr-blackhole
+tt-model push    $ROOT/build/rf-detr-blackhole --publish            # uploads code/, image/, README (generated card)
+```
 
-# 5. build, prove, publish
-tt-model package --container tt-model.yaml
-tt-model serve   build/rf-detr-blackhole/tt_kernel_manifest.json
-tt-model push    build/rf-detr-blackhole --publish
+`serve` publishes the first free port from 20000 and exports into the container exactly:
+`HF_MODEL=Roboflow/rf-detr-base`, `MESH_DEVICE=P150`, `TT_MESH_SHAPE=1x1`, then `serve.env`
+(`TT_WEIGHTS_REVISION=7b95b0…`, `TT_METAL_VISIBLE_DEVICES=0`). The HF cache is mounted at
+`/hf` (rw), kernels at `/cache`, and `/weight-cache` is unused by this port (weights are
+converted to bf16/bf8 device tensors at boot in seconds). tt-cli users:
+`tt serve changh95/rf-detr-blackhole` / `tt model stop changh95/rf-detr-blackhole`.
+
+`push` makes `code/` and `image/` on the Hub exactly the staged trees (`rf_detr`,
+`scripts`, `conftest.py` and the `models/common/lightweightmodule.py` filler) and replaces
+`README.md` with the generated card (everything worth keeping lives in
+`card.description` / `card.quickstart`). `media/`, `SERVING.md`, `.gitattributes` and
+`tt-model.yaml` at the repo root survive; the orchestrator restores the `license` /
+`pipeline_tag` front matter afterwards.
+
+## 3. The request / response contract
+
+| route | purpose |
+|---|---|
+| `GET /health` | `{"status": "ok" \| "starting", "model": "RF-DETR-base", "device": "blackhole p150 (device_id=0, mesh 1x1)" \| null}` — 200 always; `ok` only after warm-up |
+| `GET /info` | model, task, hardware, weights repo + revision + resolved files, port source + commit, input limits (560x560, 1 image, batch 1), COCO labels, device params |
+| `GET /v1/models` | `{"object": "list", "data": [{"id": "Roboflow/rf-detr-base", "object": "model", "owned_by": "changh95"}]}` — only so the OpenAI-shaped ready card / `tt-model curl` do not 404 |
+| `POST /predict` | one image -> detections (below) |
+
+`POST /predict` request (JSON):
+
+| field | type | default | meaning |
+|---|---|---|---|
+| `image` | str | required | base64 PNG/JPEG, any size (max side 8192, max 32 MB); a `data:image/...;base64,` prefix and wrapped base64 are tolerated |
+| `threshold` | float 0..1 | `0.5` | keep detections with `sigmoid(logit) > threshold` |
+| `max_detections` | int 1..300 | `100` | keep at most this many (highest scores first) |
+| `return_raw` | bool | `false` | also return `raw.logits` `[300][91]` and `raw.pred_boxes` `[300][4]` (normalised cxcywh) |
+
+Response (200):
+
+```json
+{
+  "detections": [
+    {"label": "cat", "label_id": 17, "score": 0.96,
+     "box": [7.35, 54.64, 318.45, 472.14], "box_cxcywh_norm": [0.2545, 0.5487, 0.4861, 0.8698]}
+  ],
+  "num_detections": 5,
+  "image_size": {"width": 640, "height": 480},
+  "input_size": [560, 560],
+  "threshold": 0.5,
+  "timing_ms": {"decode": 3.1, "preprocess": 4.0, "inference": 47.0, "postprocess": 0.4, "total": 54.5}
+}
 ```
 
-Step 5's `--publish` implies `--public` and adds the `tt-model-catalog` tag, which is what
-lists it in the community catalog.
-
-> **Run every `tt-model` command from this repo's root.** `source.extra_code[].root: code`
-> is resolved against the **current working directory**, not against the YAML's location --
-> verified: loading the same manifest from a parent directory fails with
-> `source.extra_code lists 1 path(s) that do not exist under code`. `cd` into the repo first.
-
-## What the bundle push does to this repo
-
-- **`code/` and `image/` are replaced wholesale** (`_prune_removed` in `tt_kernel/hub.py`).
-  The port code staged here is exactly what `extra_code` re-stages, so it is replaced, not
-  duplicated. Files under `code/` that the allowlist does not ship get pruned.
-- **`README.md` is overwritten** by the generated card. The text you want to keep belongs in
-  `card.description` / `card.quickstart` in `tt-model.yaml` -- it is already seeded there.
-- **Root files survive**, which is why `media/` and this file live at the root.
-
-## The serving contract
-
-`kind: tt-dit-server` requires only `hardware` + `mesh_device` (no `max_num_seqs` /
-`block_size` -- those are vLLM engine settings). `runtime.app` is an ASGI target
-`"module:attr"` whose **top-level package must appear in an allowlist entry**, which is why
-`runtime.app` and `source.extra_code[0].paths` have to stay in sync. The kind installs
-fastapi / uvicorn / pydantic / pillow; `runtime.packages` adds to that set.
-
-Despite the name, this kind is not diffusion-only -- `jashansinghTT/s2-pro-blackhole` is a
-TTS model using it. It simply means "the model's own ASGI app under uvicorn".
-
-## The catalog tags are automatic -- do not add them by hand
-
-`tt-model-cache`, `tt-model-container` and `tt-model-catalog` are written by the real flow,
-not by editing this card:
-
-| tag | written by | source |
-|---|---|---|
-| `tt-model-cache` | `package` | `TT_MODEL_TAG` in `tt_kernel/__init__.py:9` |
-| `tt-model-container` | `package` | `build.py:996` -- `{TT_MODEL_TAG, m.arch, m.kind, "tt-model-container"}` |
-| `blackhole`, `tt-dit-server`, `p150` | `package` | same line, from `arch` / `kind` / each profile's `hardware` |
-| `tt-model-catalog` | `--publish` | `cli.py:620` / `cli.py:820` |
-
-So finishing step 5 produces exactly the tag set the catalog models carry. Adding them to
-this repo now would be actively harmful, because **nothing validates them**:
-
-- `tt-model-cache` is what `tt-model search` filters on (`hub.py:268`), so the repo would
-  show up as an installable bundle and `tt-model pull` would then fail -- there is no
-  `tt_kernel_manifest.json` here, only the authoring `tt-model.yaml`.
-- `tt-model-container` asserts a v5.1 OCI image in `image/`, which does not exist yet.
-- `tt-model-catalog` lists it in the public community index, which renders each entry by
-  reading `tt_kernel_manifest.json`.
-- `tt-model publish <id>` does **not** check that the target is a real bundle -- it only
-  flips visibility and sets the tag -- so the guard rail you might expect is not there.
-
-Until the bundle is built, the honest tag set is the one this card already carries
-(`tenstorrent`, `blackhole`, `p150`, `ttnn`, `tt-metal`, plus the task tags).
-
-## Known sharp edges (offline-validated, not yet hardware-validated)
-
-The manifest loads clean (`load_container_manifest(..., check_sources=False)` passes, which
-includes the launcher's own `validate()`), but two things can only be settled on the box:
-
-1. **`code/` becomes the image's ONLY `models` package.** `source.code: [models/common]` is
-   staged from tt-metal into `code/models/common`, and an `extra_code` path of `models`
-   merges the port's own tree into the same `code/models/`. tt-metal's `models/` has no
-   `__init__.py`, so this relies on namespace packages resolving. Check the import in
-   `verify.sh` output on the first build.
-2. **Generically-named top-level packages.** Repos whose package is `models`, `tt`, or
-   `common` put a very common name at the root of the image's import path. If an import
-   collides, rename the package in the port repo (and update `runtime.app` +
-   `source.extra_code[].paths` together -- the launcher cross-checks them).
-
-`source.code` needs at least one tt-metal-relative entry even when all real code comes from
-`extra_code`; `models/common` is used as that minimum. If your port genuinely imports more
-from tt-metal (`locate-anything` needs `models/tt_transformers` and
-`models/demos/qwen25_vl`), list it there instead -- under-listing fails the image's own
-build-time import check on your machine, which is the cheap place to find out.
+`box` is `[x1, y1, x2, y2]` in pixels of the ORIGINAL image (the preprocessor squashes the
+whole image to 560x560, so the model's normalised cxcywh is relative to the original too;
+clamped to the image bounds), sorted by score descending. `label_id` is the COCO-91 index
+from `config.json`'s `id2label`. Errors: **400** undecodable/oversized image, **422**
+schema violation (missing `image`, `threshold` out of range), **503** while starting,
+**500** with `inference failed: <ExceptionType>: <text>` if the device call raises.
+
+Server-side pipeline (the port's validated recipe): PIL RGB -> bilinear+antialias resize
+to 560x560 -> /255 -> ImageNet mean/std -> `TtRfDetr(pixel_values)` under one global
+`threading.Lock` and `torch.inference_mode()` -> `sigmoid(logits).max(-1)` -> threshold ->
+sort -> scale cxcywh by (W, H).
+
+## 4. Caveats
+
+- **Batch 1, one image per request.** The TT graph is shape-locked to 560x560
+  (16 windows x 101 tokens, 40x40 feature grid, 300 queries, 91 classes). Requests are
+  serialised by a lock; concurrent clients queue.
+- **Warm-up captures the metal-trace** of the projector+transformer tail inside the
+  lifespan (two forwards on a synthetic 560x560 image), so READY means warm; the first
+  cold boot pays the ttnn JIT (minutes), later boots reuse `/cache` (`TT_METAL_CACHE`).
+- **Weights are pinned by sha** in both `weights.revision` and `serve.env.TT_WEIGHTS_REVISION`;
+  `hf_hub_download(..., revision=<sha>)` is a pure cache hit after `serve`'s pre-download
+  and falls back to `local_files_only=True` if the Hub is unreachable.
+- **Precision knobs stay off.** `RFDETR_BB_FIDELITY` / `RFDETR_BB_FP32ACC` / `RFDETR_BB_L1`
+  are read by `TtRfDetr` but every setting other than the default regressed accuracy or hung
+  (port README); do not put them in `serve.env`.
+- **ttnn API drift** is the main hardware-phase risk: the port was validated on a
+  ~June-2026 tree; the packaged tree is 2026-08-20. `Conv2dConfig.shard_layout`,
+  `ttnn.topk` index dtype, `execute_trace` kwargs and `grid_sample` kwargs all exist in
+  this tree (checked offline); numerics/gate (98.5 detection-IoU) must be re-confirmed
+  with `pytest code/rf_detr/tests/test_pretrained_eval.py` or the smoke test on the box.
+- The `tt_metal/python_env` venv on this host is Python 3.10 (the image uses 3.12); the
+  code is version-agnostic (3.9+ syntax, `from __future__ import annotations`).
+- `tt-model curl` and the ready card's `/v1/models` hint are OpenAI-shaped and are not this
+  API; use the routes above.