# Serving SuperPoint on Blackhole with tt-model-manager This repo is the authoring source of the **tt-model container package** `changh95/superpoint-p150` (`kind: tt-dit-server`, schema 5.1). `tt-model.yaml` is the manifest, `code/` the port, `code/models/server/app.py` the ASGI app uvicorn runs. | item | value | |---|---| | tt-metal tree | `/home/deepgadget/experiments/gbp-tt/tt-metal` (main 8b98410e730, v0.78.0-dev20260820-25, torch 2.11.0 pin) | | weights | `magic-leap-community/superpoint` @ `734450e9ffe229074f5998494ddc615475cdb20a` (`config.json`, `model.safetensors`, `preprocessor_config.json`; 5 MB; public, ungated) | | port source | github.com/changh95/tt-superpoint @ `e1eab66e29ff424bc9af6b1118671d9bc08e899e` (+ `code/models/server/`, `code/models/tt/postprocess.py` added here) | | hardware | one Blackhole p150 (`hardware: p150`, `mesh_device: P150`, `TT_MESH_SHAPE=1x1`) | | app | `models.server.app:app` | ## What the server does Default since 2026-09-13 (`TT_FUSED` unset or `1`, also pinned in `serve.env`): pure ttnn, the whole device graph as **one metal trace** per request with a standard-op device NMS, no custom kernel -- see *Fused path* below for the per-request sequence and the measured numbers. `TT_FUSED=0` restores the legacy path this section describes: pure-ttnn, **untraced**, host NMS -- the port's `SP_TRACE_NMS=0 SP_NO_TRACE=1` configuration. Both paths: fixed 480x640 network input; every image is resized server-side. Per request (legacy): 1. base64 -> PIL RGB -> bilinear resize to 640x480 -> /255 -> fp32 `(1, 3, 480, 640)` (HF `SuperPointImageProcessor` defaults; the model reads channel 0 like `SuperPointForKeypointDetection.extract_one_channel_pixel_values`). 2. `TtSuperPoint.run_untraced(tt_in, pixel_values)`: H2D into the persistent device input, `run_device_compute(trace_nms=False)` (8 convs + 3 max-pools + heads, bf16/HiFi2/fp32-acc, softmax and descriptor L2-norm on device), `device_outputs_to_host` (D2H, **no second softmax**), deallocate. 3. `models.tt.postprocess.postprocess_keypoints`: fold 8x8 cells, single-pass NMS (`nms_radius`), threshold, border removal (4 px), top-k (`max_keypoints`), bilinear descriptor sampling + L2-norm. Identical to the sequence validated in `code/models/tests/test_superpoint.py` (score PCC 0.9971, descriptor PCC 0.9991, F1 98.8%). 4. Keypoints are scaled back to the client's pixel frame (`scale = {W/640, H/480}`). One `threading.Lock` serialises every device call; handlers are sync and run under `torch.inference_mode()`. Startup (weights -> device -> model -> 2 warm-up forwards on a zero frame) happens in the ASGI lifespan, so `Application startup complete` means warm. Shutdown deallocates the device tensors and `ttnn.close_device`s inside the 120 s SIGTERM budget. Measured 2026-09-13 through the container image (50 warm requests, medians; legacy path): device_forward 12.4 ms, host NMS + post 26.5 ms, total 56.8 ms. The README's 40.7 fps requires the fused `sp_eq_mul_mask` C++ kernel (`code/kernels/`, needs a patched tt-metal) -- not what this image runs; the fused path below gets to ~40 fps with standard ops instead. ## Environment the app reads (lifespan only, never at import) | var | set by | meaning / default | |---|---|---| | `HF_MODEL` | launcher (`weights.repo`) | weights repo id; default `magic-leap-community/superpoint` | | `TT_WEIGHTS_REVISION` | `serve.env` | commit sha passed to `from_pretrained(revision=)`; default: repo default branch | | `SP_WEIGHTS_DIR` | you (host/offline) | local dir with `config.json` + `model.safetensors`; overrides the two above | | `TT_MESH_SHAPE` | launcher (`runtime.mesh_shape_env`) | `1x1` (also `(1, 1)` / `1,1`); any other shape -> `RuntimeError` at startup | | `TT_DEVICE_ID` | you | chip to open, default `0` | | `TT_FUSED` | `serve.env` (`"1"`; the code default when unset/empty is also fused) | unset/`1` = fused serving path: ONE metal trace per request (64-byte-page input upload, encoder + heads, `rms_norm` L2-norm, standard-op device NMS at radius 4, row-major outputs), captured during warm-up before READY. `0` = the legacy untraced path above, byte-identical to the 2026-09-12 image. See `DEVICE_VALIDATION.md` | | `TT_FUSED_STAGES` | you (device A/B only) | comma list of fused stages, default all (`wide,nms,rms,rm`); `""` = trace-only | | `SP_TRACE_REGION` | you | `trace_region_size` bytes for `ttnn.CreateDevice` on the fused path (default 32 MiB) | | `MESH_DEVICE` | launcher | `P150` (informational) | | `TT_METAL_VISIBLE_DEVICES` | `serve.env` | `0` | The launcher exports no revision, which is why `serve.env.TT_WEIGHTS_REVISION` repeats `weights.revision`. A sha-pinned snapshot has no `refs/main`, so the app must pass the sha (and falls back to `local_files_only=True` if the Hub is unreachable but the snapshot is cached). ## HTTP contract | route | response | |---|---| | `GET /health` | `{"status": "ok" \| "starting", "model": "superpoint-p150", "device": {"arch", "id", "open"}}` (always 200) | | `GET /info` | model/task/io, hardware, `weights {repo, revision, local_dir, loaded}`, `source {repo, commit}`, `input` (480x640, batch 1, preprocessing), `defaults`, `limits`, `serving_path` (fused: `traced=true, device_nms=true, nms_radius_traced=4, fused_stages`; legacy: `traced=false, device_nms=false`), `warmup_ms`, `descriptors` encoding, `license` | | `GET /v1/models` | `{"object": "list", "data": [{"id": "", "object": "model", "owned_by": "changh95"}]}` (so OpenAI-shaped probes do not 404; not a chat API) | | `POST /predict` | see below | Request (`application/json`): ```json { "image": "", // required; RGB or grayscale; any size (resized to 640x480) "max_keypoints": 1024, // optional; -1 = all above threshold; cap 307200 "keypoint_threshold": 0.005, // optional; [0, 1] "nms_radius": 4, // optional; 0..32 canonical-frame pixels; 0 = no NMS "return_descriptors": true // optional } ``` Response: ```json { "num_keypoints": N, "keypoints": [[x, y], ...], // N x 2 floats, ORIGINAL image pixel coordinates "scores": [...], // N floats, always descending (keypoints/descriptors share the order) "original_size": {"height": H, "width": W}, "image_size": {"height": 480, "width": 640}, "scale": {"x": W/640, "y": H/480}, // divide keypoints by this to get network-frame coords "params": {"max_keypoints", "keypoint_threshold", "nms_radius", "border_removal_distance"}, "timing_ms": {"preprocess", "device_forward", "postprocess", "total"}, "descriptors": { // only when return_descriptors is true "format": "npz", "key": "descriptors", "dtype": "float16", "shape": [N, 256], "data": "" // numpy.load(io.BytesIO(base64.b64decode(data)))["descriptors"] } } ``` Errors: `400` undecodable image or invalid field (pydantic errors are mapped to 400), `503` while starting, `500` with `": "` on an inference failure. One image per request. Smoke test (the hardware phase runs it unchanged): ```bash python code/models/server/smoke_test.py --url http://127.0.0.1: # PASS superpoint-p150: keypoints on house_in_field_1080p.jpg (1600x900), ... ``` ## Running on the HOST for validation (no Docker) Uses the tree's own venv (`python_env`, Python 3.10, torch 2.11.0+cpu, transformers 5.12.1) plus fastapi/uvicorn, which that venv lacks -- install them into a throwaway venv and append its `site-packages` rather than touching the tree venv: ```bash ROOT=/home/deepgadget/experiments/tt-models T=/home/deepgadget/experiments/gbp-tt/tt-metal export PATH=$HOME/.local/bin:$PATH uv venv --python 3.10 /tmp/sp-http -q && uv pip install --python /tmp/sp-http/bin/python -q fastapi uvicorn export PYTHONPATH=$ROOT/models/superpoint-p150/code:$T:$T/ttnn:$T/tools:/tmp/sp-http/lib/python3.10/site-packages export TT_METAL_HOME=$T ARCH_NAME=blackhole export HF_MODEL=magic-leap-community/superpoint export TT_WEIGHTS_REVISION=734450e9ffe229074f5998494ddc615475cdb20a export TT_MESH_SHAPE=1x1 TT_DEVICE_ID=0 TT_METAL_VISIBLE_DEVICES=0 # import check with NO device (what the image's verify.sh does): cd /tmp && $T/python_env/bin/python -c "import models.server.app as a; assert a.app" # serve (opens the chip; hardware phase only): cd $ROOT/models/superpoint-p150 && $T/python_env/bin/python -m uvicorn --host 0.0.0.0 --port 20000 --lifespan on models.server.app:app # then, from another shell: python code/models/server/smoke_test.py --url http://127.0.0.1:20000 ``` Boot log landmarks: `Loading weights ...`, `Opening device 0`, `Warming up (compiling kernels ...)`, `Warmup complete: first forward (compile), second `, then uvicorn's `Application startup complete`. Ctrl-C / SIGTERM closes the device. `PYTHONPATH` must start with `code/` so that `models` resolves to this repo's regular package (tt-metal's own `models/` has no `__init__.py` and is shadowed on the host; in the image it is excluded). The device tests in `code/models/tests/test_superpoint.py` get their `device` / `device_params` fixtures and `--device-id` from the repo-local `code/conftest.py` (run pytest from `code/`; `code/pytest.ini` pins the rootdir). tt-metal's own conftest cannot be used next to this repo: it imports `models.demos...`, which the shadowing above breaks. ## Package / serve / push (Blackhole host, rootless Docker) Every command from **this directory**: `extra_code.root: code` and `--out` resolve against the CWD. `source $ROOT/bin/docker-env.sh` first (rootless Docker 28 + buildx on this box; the bare `docker` on PATH is podman). ```bash ROOT=/home/deepgadget/experiments/tt-models cd $ROOT/models/superpoint-p150 source $ROOT/bin/docker-env.sh # offline validation (no docker, no device) -- must print VALID $ROOT/.venv/bin/python -c "from tt_kernel.container_manifest import load_container_manifest; m = load_container_manifest('tt-model.yaml', check_sources=True); p = m.resolve_profile(); print('VALID', m.name, m.kind, p.hardware, p.mesh_device, m.weights_ref)" # build (2.5-4 h cold; verify.sh imports the app + the verify: lines inside the image) $ROOT/.venv/bin/tt-model package --container tt-model.yaml --out $ROOT/build # serve + smoke (hardware phase) $ROOT/.venv/bin/tt-model serve $ROOT/build/superpoint-p150/tt_kernel_manifest.json python code/models/server/smoke_test.py --url http://127.0.0.1: $ROOT/.venv/bin/tt-model stop changh95/superpoint-p150 # publish (after validation only) $ROOT/.venv/bin/tt-model push $ROOT/build/superpoint-p150 --publish ``` tt-cli users: `tt serve changh95/superpoint-p150` / `tt model stop changh95/superpoint-p150` (`tt config set tools.override.tt-model $ROOT/.venv/bin/tt-model` is already set here because the pinned tt-model breaks under rootless Docker). ## What push does to this repo `code/` and `image/` on the Hub become exactly the staged trees, so `extra_code.paths` lists everything under `code/` worth keeping: `models`, `kernels` (fused NMS kernel source, not built into this image), `sample_data`, `results.tsv`, `run_benchmark.sh`. `code/.gitignore` is not listed and will be pruned. `README.md` is replaced by the generated card (all README content worth keeping lives in `card.description` / `card.quickstart`); `media/`, `SERVING.md`, `.gitattributes` and `tt-model.yaml` at the root survive. The orchestrator restores `license`/`pipeline_tag` front matter after push. ## Fused path (default; `TT_FUSED=0` = legacy) -- branch `opt/superpoint-p150-megakernel` Device-validated on the p150a 2026-09-13 (`DEVICE_VALIDATION.md` "Results") and made the default (code default + `serve.env.TT_FUSED: "1"`); `TT_FUSED=0` restores the legacy path above byte-for-byte. On the fused path the lifespan opens the device with a trace region, runs the fused graph once eagerly (kernel compile), captures it into a metal trace and replays it once -- all before `Application startup complete` (boot log: `Warming up TT_FUSED path ...`, `Warmup complete: compile forward ... trace capture ... traced forward ...`). Per request: H2D of the `[1,1,9600,32]` bf16 input, one `execute_trace`, D2H of the row-major device NMS map (480x640) and descriptors, then threshold / border / top-k / `grid_sample` on the host (`postprocess_from_nms_map`). Requests with `nms_radius != 4` read the traced softmax scores instead and run the legacy host NMS (same output, slower); `response.serving_path.device_nms` and `/info.serving_path` report which path ran. Exactness: trace, upload, device NMS and row-major outputs are bit-identical to the legacy path; the `rms_norm` descriptor L2-norm is bf16-rounding-level (gate: descriptor PCC >= 0.999). Host proofs: `code/models/tests/test_fused_host.py`; hardware plan, gates and results: `DEVICE_VALIDATION.md`. Measured 2026-09-13 through the container image (`smoke_test.py` PASS, 50 warm requests, server `timing_ms` medians): device_forward 5.3 ms (min 5.1), post-processing 1.3 ms, preprocess 18 ms, total 24.7 ms (~40 fps) vs the legacy path's 12.4 / 26.5 / 17.8 / 56.8 ms in the same session; keypoints and scores identical to the legacy server for `nms_radius` 4 (device NMS) and 3 (host fallback), descriptors within bf16 rounding (max |diff| 8.5e-4, cosine >= 0.999996). Boot log landmarks: `Warming up TT_FUSED path (stages nms,rm,rms,wide, traced nms_radius 4)`, `Warmup complete: compile forward ... trace capture ... traced forward ...`. ## Caveats - **Licence**: the weights are Magic Leap "academic or non-profit organisation noncommercial research use only". Stated in the card; publishing to the public catalog inherits it. - **Package name `models`** collides with tt-metal's `models/` tree. It works because the image excludes tt-metal's `models/` and the port ships `models/__init__.py`; the filler `models/common/lightweightmodule.py` becomes a namespace subpackage nobody imports. - **API drift**: the port was validated on a tt-metal of ~v0.71 (spring 2026); this package builds against v0.78 (Aug 2026). `import ttnn` and every symbol the port touches (`Conv2dSliceConfig`, `Conv2dDRAMSliceHeight`, `UnaryWithParam`, `CreateDevice`, `copy_host_to_device_tensor(cq_id=)`, `max_pool2d` kwargs) exist in the v0.78 tree, but conv/pool behaviour (L1 budgets of the `(4, 2, 1, 1)` DRAM slicing) is only proven on the chip -- re-run the PCC benchmark in the hardware phase. - **Device open**: `ttnn.CreateDevice(device_id, l1_small_size=32768)` -- the recipe of the port's untraced script (`models/visualize.py`); no trace region, one command queue. - **Thread model**: uvicorn runs the sync `predict` in a worker thread; the lock serialises ttnn calls, and the lifespan opened the device on the event-loop thread (same pattern as the template and other tt-dit servers). - **Host venv is Python 3.10**, the image is 3.12 -- the uv dry-run of the runtime packages on 3.12 resolves to torch 2.11.0+cpu, numpy 1.26.4, transformers 5.17.0, no torchvision. - `transformers` in the image resolves to 5.x (5.17.0 at authoring time); `SuperPointForKeypointDetection` exists in 4.53 .. 5.17 (`>=4.53,<6` pin).