changh95
/

superpoint-p150

@@ -1,112 +1,206 @@
-# Serving this port with tt-model-manager
-This repo carries everything needed to build a **tt-model container bundle** from it, so
-`tt-model pull` / `tt-model serve` and `tt-cli` can run it. Do this on a
-**Blackhole host** (amd64 Linux + Docker >= 25).
-## Why not on a Mac
-`tt-model package --container` builds from `ghcr.io/.../ubuntu-22.04-dev-amd64` and a cold
-build is 2.5-4 hours. Packaging and pushing need **no card** (`require_host(need_devices=False)`),
-but they do need amd64. `tt-model serve` needs `/dev/tenstorrent/*`.
-## Steps
 ```bash
-# 0. get tt-model (not on PyPI)
-git clone https://github.com/tenstorrent/tt-model-manager && cd tt-model-manager
-python -m venv .venv && source .venv/bin/activate && pip install -e .
-# 1. pull THIS repo
-hf download changh95/superpoint-blackhole --local-dir superpoint-blackhole && cd superpoint-blackhole
-# 2. point the manifest at your tt-metal checkout
-$EDITOR tt-model.yaml          # source.tt_metal: /path/to/tt-metal
-# 3. finish the serving adapter
-$EDITOR code/models/server/app.py   # see "VERIFY ON HARDWARE" markers
-# 4. validate with no hardware and no build
-python -c "from tt_kernel.container_manifest import ContainerManifest as M; M.load('tt-model.yaml')"
-# 5. build, prove, publish
-tt-model package --container tt-model.yaml
-tt-model serve   build/superpoint-blackhole/tt_kernel_manifest.json
-tt-model push    build/superpoint-blackhole --publish
 ```
-Step 5's `--publish` implies `--public` and adds the `tt-model-catalog` tag, which is what
-lists it in the community catalog.
-> **Run every `tt-model` command from this repo's root.** `source.extra_code[].root: code`
-> is resolved against the **current working directory**, not against the YAML's location --
-> verified: loading the same manifest from a parent directory fails with
-> `source.extra_code lists 1 path(s) that do not exist under code`. `cd` into the repo first.
-## What the bundle push does to this repo
-- **`code/` and `image/` are replaced wholesale** (`_prune_removed` in `tt_kernel/hub.py`).
-  The port code staged here is exactly what `extra_code` re-stages, so it is replaced, not
-  duplicated. Files under `code/` that the allowlist does not ship get pruned.
-- **`README.md` is overwritten** by the generated card. The text you want to keep belongs in
-  `card.description` / `card.quickstart` in `tt-model.yaml` -- it is already seeded there.
-- **Root files survive**, which is why `media/` and this file live at the root.
-## The serving contract
-`kind: tt-dit-server` requires only `hardware` + `mesh_device` (no `max_num_seqs` /
-`block_size` -- those are vLLM engine settings). `runtime.app` is an ASGI target
-`"module:attr"` whose **top-level package must appear in an allowlist entry**, which is why
-`runtime.app` and `source.extra_code[0].paths` have to stay in sync. The kind installs
-fastapi / uvicorn / pydantic / pillow; `runtime.packages` adds to that set.
-Despite the name, this kind is not diffusion-only -- `jashansinghTT/s2-pro-blackhole` is a
-TTS model using it. It simply means "the model's own ASGI app under uvicorn".
-## The catalog tags are automatic -- do not add them by hand
-`tt-model-cache`, `tt-model-container` and `tt-model-catalog` are written by the real flow,
-not by editing this card:
-| tag | written by | source |
-|---|---|---|
-| `tt-model-cache` | `package` | `TT_MODEL_TAG` in `tt_kernel/__init__.py:9` |
-| `tt-model-container` | `package` | `build.py:996` -- `{TT_MODEL_TAG, m.arch, m.kind, "tt-model-container"}` |
-| `blackhole`, `tt-dit-server`, `p150` | `package` | same line, from `arch` / `kind` / each profile's `hardware` |
-| `tt-model-catalog` | `--publish` | `cli.py:620` / `cli.py:820` |
-So finishing step 5 produces exactly the tag set the catalog models carry. Adding them to
-this repo now would be actively harmful, because **nothing validates them**:
-- `tt-model-cache` is what `tt-model search` filters on (`hub.py:268`), so the repo would
-  show up as an installable bundle and `tt-model pull` would then fail -- there is no
-  `tt_kernel_manifest.json` here, only the authoring `tt-model.yaml`.
-- `tt-model-container` asserts a v5.1 OCI image in `image/`, which does not exist yet.
-- `tt-model-catalog` lists it in the public community index, which renders each entry by
-  reading `tt_kernel_manifest.json`.
-- `tt-model publish <id>` does **not** check that the target is a real bundle -- it only
-  flips visibility and sets the tag -- so the guard rail you might expect is not there.
-Until the bundle is built, the honest tag set is the one this card already carries
-(`tenstorrent`, `blackhole`, `p150`, `ttnn`, `tt-metal`, plus the task tags).
-## Known sharp edges (offline-validated, not yet hardware-validated)
-The manifest loads clean (`load_container_manifest(..., check_sources=False)` passes, which
-includes the launcher's own `validate()`), but two things can only be settled on the box:
-1. **`code/` becomes the image's ONLY `models` package.** `source.code: [models/common]` is
-   staged from tt-metal into `code/models/common`, and an `extra_code` path of `models`
-   merges the port's own tree into the same `code/models/`. tt-metal's `models/` has no
-   `__init__.py`, so this relies on namespace packages resolving. Check the import in
-   `verify.sh` output on the first build.
-2. **Generically-named top-level packages.** Repos whose package is `models`, `tt`, or
-   `common` put a very common name at the root of the image's import path. If an import
-   collides, rename the package in the port repo (and update `runtime.app` +
-   `source.extra_code[].paths` together -- the launcher cross-checks them).
-`source.code` needs at least one tt-metal-relative entry even when all real code comes from
-`extra_code`; `models/common` is used as that minimum. If your port genuinely imports more
-from tt-metal (`locate-anything` needs `models/tt_transformers` and
-`models/demos/qwen25_vl`), list it there instead -- under-listing fails the image's own
-build-time import check on your machine, which is the cheap place to find out.

+# Serving SuperPoint on Blackhole with tt-model-manager
+This repo is the authoring source of the **tt-model container package**
+`changh95/superpoint-blackhole` (`kind: tt-dit-server`, schema 5.1). `tt-model.yaml` is
+the manifest, `code/` the port, `code/models/server/app.py` the ASGI app uvicorn runs.
+| item | value |
+|---|---|
+| tt-metal tree | `/home/deepgadget/experiments/gbp-tt/tt-metal` (main 8b98410e730, v0.78.0-dev20260820-25, torch 2.11.0 pin) |
+| weights | `magic-leap-community/superpoint` @ `734450e9ffe229074f5998494ddc615475cdb20a` (`config.json`, `model.safetensors`, `preprocessor_config.json`; 5 MB; public, ungated) |
+| port source | github.com/changh95/tt-superpoint @ `e1eab66e29ff424bc9af6b1118671d9bc08e899e` (+ `code/models/server/`, `code/models/tt/postprocess.py` added here) |
+| hardware | one Blackhole p150 (`hardware: p150`, `mesh_device: P150`, `TT_MESH_SHAPE=1x1`) |
+| app | `models.server.app:app` |
+## What the server does
+Pure-ttnn, **untraced**, host NMS -- the port's `SP_TRACE_NMS=0 SP_NO_TRACE=1` configuration,
+no custom kernel. Fixed 480x640 network input; every image is resized server-side. Per request:
+1. base64 -> PIL RGB -> bilinear resize to 640x480 -> /255 -> fp32 `(1, 3, 480, 640)`
+   (HF `SuperPointImageProcessor` defaults; the model reads channel 0 like
+   `SuperPointForKeypointDetection.extract_one_channel_pixel_values`).
+2. `TtSuperPoint.run_untraced(tt_in, pixel_values)`: H2D into the persistent device input,
+   `run_device_compute(trace_nms=False)` (8 convs + 3 max-pools + heads, bf16/HiFi2/fp32-acc,
+   softmax and descriptor L2-norm on device), `device_outputs_to_host` (D2H, **no second
+   softmax**), deallocate.
+3. `models.tt.postprocess.postprocess_keypoints`: fold 8x8 cells, single-pass NMS
+   (`nms_radius`), threshold, border removal (4 px), top-k (`max_keypoints`), bilinear
+   descriptor sampling + L2-norm. Identical to the sequence validated in
+   `code/models/tests/test_superpoint.py` (score PCC 0.9971, descriptor PCC 0.9991, F1 98.8%).
+4. Keypoints are scaled back to the client's pixel frame (`scale = {W/640, H/480}`).
+One `threading.Lock` serialises every device call; handlers are sync and run under
+`torch.inference_mode()`. Startup (weights -> device -> model -> 2 warm-up forwards on a zero
+frame) happens in the ASGI lifespan, so `Application startup complete` means warm. Shutdown
+deallocates the device tensors and `ttnn.close_device`s inside the 120 s SIGTERM budget.
+Expected speed: ~6 fps device forward (untraced) + ~36 ms host NMS. The README's 40.7 fps
+requires trace capture and the fused `sp_eq_mul_mask` C++ kernel (`code/kernels/`, needs a
+patched tt-metal) -- deliberately not what this image runs.
+## Environment the app reads (lifespan only, never at import)
+| var | set by | meaning / default |
+|---|---|---|
+| `HF_MODEL` | launcher (`weights.repo`) | weights repo id; default `magic-leap-community/superpoint` |
+| `TT_WEIGHTS_REVISION` | `serve.env` | commit sha passed to `from_pretrained(revision=)`; default: repo default branch |
+| `SP_WEIGHTS_DIR` | you (host/offline) | local dir with `config.json` + `model.safetensors`; overrides the two above |
+| `TT_MESH_SHAPE` | launcher (`runtime.mesh_shape_env`) | `1x1` (also `(1, 1)` / `1,1`); any other shape -> `RuntimeError` at startup |
+| `TT_DEVICE_ID` | you | chip to open, default `0` |
+| `MESH_DEVICE` | launcher | `P150` (informational) |
+| `TT_METAL_VISIBLE_DEVICES` | `serve.env` | `0` |
+The launcher exports no revision, which is why `serve.env.TT_WEIGHTS_REVISION` repeats
+`weights.revision`. A sha-pinned snapshot has no `refs/main`, so the app must pass the sha
+(and falls back to `local_files_only=True` if the Hub is unreachable but the snapshot is cached).
+## HTTP contract
+| route | response |
+|---|---|
+| `GET /health` | `{"status": "ok" \| "starting", "model": "superpoint-blackhole", "device": {"arch", "id", "open"}}` (always 200) |
+| `GET /info` | model/task/io, hardware, `weights {repo, revision, local_dir, loaded}`, `source {repo, commit}`, `input` (480x640, batch 1, preprocessing), `defaults`, `limits`, `serving_path` (traced=false, device_nms=false), `warmup_ms`, `descriptors` encoding, `license` |
+| `GET /v1/models` | `{"object": "list", "data": [{"id": "<weights repo>", "object": "model", "owned_by": "changh95"}]}` (so OpenAI-shaped probes do not 404; not a chat API) |
+| `POST /predict` | see below |
+Request (`application/json`):
+```json
+{
+  "image": "<base64 PNG/JPEG>",       // required; RGB or grayscale; any size (resized to 640x480)
+  "max_keypoints": 1024,              // optional; -1 = all above threshold; cap 307200
+  "keypoint_threshold": 0.005,        // optional; [0, 1]
+  "nms_radius": 4,                    // optional; 0..32 canonical-frame pixels; 0 = no NMS
+  "return_descriptors": true          // optional
+}
+```
+Response:
+```json
+{
+  "num_keypoints": N,
+  "keypoints": [[x, y], ...],         // N x 2 floats, ORIGINAL image pixel coordinates
+  "scores": [...],                    // N floats, always descending (keypoints/descriptors share the order)
+  "original_size": {"height": H, "width": W},
+  "image_size": {"height": 480, "width": 640},
+  "scale": {"x": W/640, "y": H/480},  // divide keypoints by this to get network-frame coords
+  "params": {"max_keypoints", "keypoint_threshold", "nms_radius", "border_removal_distance"},
+  "timing_ms": {"preprocess", "device_forward", "postprocess", "total"},
+  "descriptors": {                    // only when return_descriptors is true
+    "format": "npz", "key": "descriptors", "dtype": "float16", "shape": [N, 256],
+    "data": "<base64 NPZ>"            // numpy.load(io.BytesIO(base64.b64decode(data)))["descriptors"]
+  }
+}
+```
+Errors: `400` undecodable image or invalid field (pydantic errors are mapped to 400),
+`503` while starting, `500` with `"<ExceptionType>: <message>"` on an inference failure.
+One image per request.
+Smoke test (the hardware phase runs it unchanged):
 ```bash
+python code/models/server/smoke_test.py --url http://127.0.0.1:<port>
+# PASS superpoint-blackhole: <N> keypoints on house_in_field_1080p.jpg (1600x900), ...
+```
+## Running on the HOST for validation (no Docker)
+Uses the tree's own venv (`python_env`, Python 3.10, torch 2.11.0+cpu, transformers 5.12.1)
+plus fastapi/uvicorn, which that venv lacks -- install them into a throwaway venv and append
+its `site-packages` rather than touching the tree venv:
+```bash
+ROOT=/home/deepgadget/experiments/tt-models
+T=/home/deepgadget/experiments/gbp-tt/tt-metal
+export PATH=$HOME/.local/bin:$PATH
+uv venv --python 3.10 /tmp/sp-http -q && uv pip install --python /tmp/sp-http/bin/python -q fastapi uvicorn
+export PYTHONPATH=$ROOT/models/superpoint-blackhole/code:$T:$T/ttnn:$T/tools:/tmp/sp-http/lib/python3.10/site-packages
+export TT_METAL_HOME=$T ARCH_NAME=blackhole
+export HF_MODEL=magic-leap-community/superpoint
+export TT_WEIGHTS_REVISION=734450e9ffe229074f5998494ddc615475cdb20a
+export TT_MESH_SHAPE=1x1 TT_DEVICE_ID=0 TT_METAL_VISIBLE_DEVICES=0
+# import check with NO device (what the image's verify.sh does):
+cd /tmp && $T/python_env/bin/python -c "import models.server.app as a; assert a.app"
+# serve (opens the chip; hardware phase only):
+cd $ROOT/models/superpoint-blackhole && $T/python_env/bin/python -m uvicorn --host 0.0.0.0 --port 20000 --lifespan on models.server.app:app
+# then, from another shell:
+python code/models/server/smoke_test.py --url http://127.0.0.1:20000
 ```
+Boot log landmarks: `Loading weights ...`, `Opening device 0`, `Warming up (compiling
+kernels ...)`, `Warmup complete: first forward <ms> (compile), second <ms>`, then uvicorn's
+`Application startup complete`. Ctrl-C / SIGTERM closes the device.
+`PYTHONPATH` must start with `code/` so that `models` resolves to this repo's regular package
+(tt-metal's own `models/` has no `__init__.py` and is shadowed on the host; in the image it is
+excluded). The benchmark `code/models/tests/test_superpoint.py` still needs a tt-metal checkout as
+pytest rootdir for its `device` fixture -- it is unchanged by the packaging work.
+## Package / serve / push (Blackhole host, rootless Docker)
+Every command from **this directory**: `extra_code.root: code` and `--out` resolve against the
+CWD. `source $ROOT/bin/docker-env.sh` first (rootless Docker 28 + buildx on this box; the bare
+`docker` on PATH is podman).
+```bash
+ROOT=/home/deepgadget/experiments/tt-models
+cd $ROOT/models/superpoint-blackhole
+source $ROOT/bin/docker-env.sh
+# offline validation (no docker, no device) -- must print VALID
+$ROOT/.venv/bin/python -c "from tt_kernel.container_manifest import load_container_manifest; m = load_container_manifest('tt-model.yaml', check_sources=True); p = m.resolve_profile(); print('VALID', m.name, m.kind, p.hardware, p.mesh_device, m.weights_ref)"
+# build (2.5-4 h cold; verify.sh imports the app + the verify: lines inside the image)
+$ROOT/.venv/bin/tt-model package --container tt-model.yaml --out $ROOT/build
+# serve + smoke (hardware phase)
+$ROOT/.venv/bin/tt-model serve $ROOT/build/superpoint-blackhole/tt_kernel_manifest.json
+python code/models/server/smoke_test.py --url http://127.0.0.1:<port serve printed>
+$ROOT/.venv/bin/tt-model stop changh95/superpoint-blackhole
+# publish (after validation only)
+$ROOT/.venv/bin/tt-model push $ROOT/build/superpoint-blackhole --publish
+```
+tt-cli users: `tt serve changh95/superpoint-blackhole` / `tt model stop changh95/superpoint-blackhole`
+(`tt config set tools.override.tt-model $ROOT/.venv/bin/tt-model` is already set here because the
+pinned tt-model breaks under rootless Docker).
+## What push does to this repo
+`code/` and `image/` on the Hub become exactly the staged trees, so `extra_code.paths` lists
+everything under `code/` worth keeping: `models`, `kernels` (fused NMS kernel source, not built
+into this image), `sample_data`, `results.tsv`, `run_benchmark.sh`. `code/.gitignore` is not
+listed and will be pruned. `README.md` is replaced by the generated card (all README content
+worth keeping lives in `card.description` / `card.quickstart`); `media/`, `SERVING.md`,
+`.gitattributes` and `tt-model.yaml` at the root survive. The orchestrator restores
+`license`/`pipeline_tag` front matter after push.
+## Caveats
+- **Licence**: the weights are Magic Leap "academic or non-profit organisation noncommercial
+  research use only". Stated in the card; publishing to the public catalog inherits it.
+- **Package name `models`** collides with tt-metal's `models/` tree. It works because the
+  image excludes tt-metal's `models/` and the port ships `models/__init__.py`; the filler
+  `models/common/lightweightmodule.py` becomes a namespace subpackage nobody imports.
+- **API drift**: the port was validated on a tt-metal of ~v0.71 (spring 2026); this package
+  builds against v0.78 (Aug 2026). `import ttnn` and every symbol the port touches
+  (`Conv2dSliceConfig`, `Conv2dDRAMSliceHeight`, `UnaryWithParam`, `CreateDevice`,
+  `copy_host_to_device_tensor(cq_id=)`, `max_pool2d` kwargs) exist in the v0.78 tree, but
+  conv/pool behaviour (L1 budgets of the `(4, 2, 1, 1)` DRAM slicing) is only proven on the
+  chip -- re-run the PCC benchmark in the hardware phase.
+- **Device open**: `ttnn.CreateDevice(device_id, l1_small_size=32768)` -- the recipe of the
+  port's untraced script (`models/visualize.py`); no trace region, one command queue.
+- **Thread model**: uvicorn runs the sync `predict` in a worker thread; the lock serialises
+  ttnn calls, and the lifespan opened the device on the event-loop thread (same pattern as the
+  template and other tt-dit servers).
+- **Host venv is Python 3.10**, the image is 3.12 -- the uv dry-run of the runtime packages
+  on 3.12 resolves to torch 2.11.0+cpu, numpy 1.26.4, transformers 5.17.0, no torchvision.
+- `transformers` in the image resolves to 5.x (5.17.0 at authoring time); `SuperPointForKeypointDetection`
+  exists in 4.53 .. 5.17 (`>=4.53,<6` pin).