Serving SuperPoint on Blackhole with tt-model-manager

This repo is the authoring source of the tt-model container package changh95/superpoint-p150 (kind: tt-dit-server, schema 5.1). tt-model.yaml is the manifest, code/ the port, code/models/server/app.py the ASGI app uvicorn runs.

item	value
tt-metal tree	`/home/deepgadget/experiments/gbp-tt/tt-metal` (main 8b98410e730, v0.78.0-dev20260820-25, torch 2.11.0 pin)
weights	`magic-leap-community/superpoint` @ `734450e9ffe229074f5998494ddc615475cdb20a` (`config.json`, `model.safetensors`, `preprocessor_config.json`; 5 MB; public, ungated)
port source	github.com/changh95/tt-superpoint @ `e1eab66e29ff424bc9af6b1118671d9bc08e899e` (+ `code/models/server/`, `code/models/tt/postprocess.py` added here)
hardware	one Blackhole p150 (`hardware: p150`, `mesh_device: P150`, `TT_MESH_SHAPE=1x1`)
app	`models.server.app:app`

What the server does

Default since 2026-09-13 (TT_FUSED unset or 1, also pinned in serve.env): pure ttnn, the whole device graph as one metal trace per request with a standard-op device NMS, no custom kernel -- see Fused path below for the per-request sequence and the measured numbers. TT_FUSED=0 restores the legacy path this section describes: pure-ttnn, untraced, host NMS -- the port's SP_TRACE_NMS=0 SP_NO_TRACE=1 configuration. Both paths: fixed 480x640 network input; every image is resized server-side. Per request (legacy):

base64 -> PIL RGB -> bilinear resize to 640x480 -> /255 -> fp32 (1, 3, 480, 640) (HF SuperPointImageProcessor defaults; the model reads channel 0 like SuperPointForKeypointDetection.extract_one_channel_pixel_values).
TtSuperPoint.run_untraced(tt_in, pixel_values): H2D into the persistent device input, run_device_compute(trace_nms=False) (8 convs + 3 max-pools + heads, bf16/HiFi2/fp32-acc, softmax and descriptor L2-norm on device), device_outputs_to_host (D2H, no second softmax), deallocate.
models.tt.postprocess.postprocess_keypoints: fold 8x8 cells, single-pass NMS (nms_radius), threshold, border removal (4 px), top-k (max_keypoints), bilinear descriptor sampling + L2-norm. Identical to the sequence validated in code/models/tests/test_superpoint.py (score PCC 0.9971, descriptor PCC 0.9991, F1 98.8%).
Keypoints are scaled back to the client's pixel frame (scale = {W/640, H/480}).

One threading.Lock serialises every device call; handlers are sync and run under torch.inference_mode(). Startup (weights -> device -> model -> 2 warm-up forwards on a zero frame) happens in the ASGI lifespan, so Application startup complete means warm. Shutdown deallocates the device tensors and ttnn.close_devices inside the 120 s SIGTERM budget.

Measured 2026-09-13 through the container image (50 warm requests, medians; legacy path): device_forward 12.4 ms, host NMS + post 26.5 ms, total 56.8 ms. The README's 40.7 fps requires the fused sp_eq_mul_mask C++ kernel (code/kernels/, needs a patched tt-metal) -- not what this image runs; the fused path below gets to ~40 fps with standard ops instead.

Environment the app reads (lifespan only, never at import)

var	set by	meaning / default
`HF_MODEL`	launcher (`weights.repo`)	weights repo id; default `magic-leap-community/superpoint`
`TT_WEIGHTS_REVISION`	`serve.env`	commit sha passed to `from_pretrained(revision=)`; default: repo default branch
`SP_WEIGHTS_DIR`	you (host/offline)	local dir with `config.json` + `model.safetensors`; overrides the two above
`TT_MESH_SHAPE`	launcher (`runtime.mesh_shape_env`)	`1x1` (also `(1, 1)` / `1,1`); any other shape -> `RuntimeError` at startup
`TT_DEVICE_ID`	you	chip to open, default `0`
`TT_FUSED`	`serve.env` (`"1"`; the code default when unset/empty is also fused)	unset/`1` = fused serving path: ONE metal trace per request (64-byte-page input upload, encoder + heads, `rms_norm` L2-norm, standard-op device NMS at radius 4, row-major outputs), captured during warm-up before READY. `0` = the legacy untraced path above, byte-identical to the 2026-09-12 image. See `DEVICE_VALIDATION.md`
`TT_FUSED_STAGES`	you (device A/B only)	comma list of fused stages, default all (`wide,nms,rms,rm`); `""` = trace-only
`SP_TRACE_REGION`	you	`trace_region_size` bytes for `ttnn.CreateDevice` on the fused path (default 32 MiB)
`MESH_DEVICE`	launcher	`P150` (informational)
`TT_METAL_VISIBLE_DEVICES`	`serve.env`	`0`

The launcher exports no revision, which is why serve.env.TT_WEIGHTS_REVISION repeats weights.revision. A sha-pinned snapshot has no refs/main, so the app must pass the sha (and falls back to local_files_only=True if the Hub is unreachable but the snapshot is cached).

HTTP contract

route	response
`GET /health`	`{"status": "ok" \| "starting", "model": "superpoint-p150", "device": {"arch", "id", "open"}}` (always 200)
`GET /info`	model/task/io, hardware, `weights {repo, revision, local_dir, loaded}`, `source {repo, commit}`, `input` (480x640, batch 1, preprocessing), `defaults`, `limits`, `serving_path` (fused: `traced=true, device_nms=true, nms_radius_traced=4, fused_stages`; legacy: `traced=false, device_nms=false`), `warmup_ms`, `descriptors` encoding, `license`
`GET /v1/models`	`{"object": "list", "data": [{"id": "<weights repo>", "object": "model", "owned_by": "changh95"}]}` (so OpenAI-shaped probes do not 404; not a chat API)
`POST /predict`	see below

Request (application/json):

{
  "image": "<base64 PNG/JPEG>",       // required; RGB or grayscale; any size (resized to 640x480)
  "max_keypoints": 1024,              // optional; -1 = all above threshold; cap 307200
  "keypoint_threshold": 0.005,        // optional; [0, 1]
  "nms_radius": 4,                    // optional; 0..32 canonical-frame pixels; 0 = no NMS
  "return_descriptors": true          // optional
}

Response:

{
  "num_keypoints": N,
  "keypoints": [[x, y], ...],         // N x 2 floats, ORIGINAL image pixel coordinates
  "scores": [...],                    // N floats, always descending (keypoints/descriptors share the order)
  "original_size": {"height": H, "width": W},
  "image_size": {"height": 480, "width": 640},
  "scale": {"x": W/640, "y": H/480},  // divide keypoints by this to get network-frame coords
  "params": {"max_keypoints", "keypoint_threshold", "nms_radius", "border_removal_distance"},
  "timing_ms": {"preprocess", "device_forward", "postprocess", "total"},
  "descriptors": {                    // only when return_descriptors is true
    "format": "npz", "key": "descriptors", "dtype": "float16", "shape": [N, 256],
    "data": "<base64 NPZ>"            // numpy.load(io.BytesIO(base64.b64decode(data)))["descriptors"]
  }
}

Errors: 400 undecodable image or invalid field (pydantic errors are mapped to 400), 503 while starting, 500 with "<ExceptionType>: <message>" on an inference failure. One image per request.

Smoke test (the hardware phase runs it unchanged):

python code/models/server/smoke_test.py --url http://127.0.0.1:<port>
# PASS superpoint-p150: <N> keypoints on house_in_field_1080p.jpg (1600x900), ...

Running on the HOST for validation (no Docker)

Uses the tree's own venv (python_env, Python 3.10, torch 2.11.0+cpu, transformers 5.12.1) plus fastapi/uvicorn, which that venv lacks -- install them into a throwaway venv and append its site-packages rather than touching the tree venv:

ROOT=/home/deepgadget/experiments/tt-models
T=/home/deepgadget/experiments/gbp-tt/tt-metal
export PATH=$HOME/.local/bin:$PATH
uv venv --python 3.10 /tmp/sp-http -q && uv pip install --python /tmp/sp-http/bin/python -q fastapi uvicorn

export PYTHONPATH=$ROOT/models/superpoint-p150/code:$T:$T/ttnn:$T/tools:/tmp/sp-http/lib/python3.10/site-packages
export TT_METAL_HOME=$T ARCH_NAME=blackhole
export HF_MODEL=magic-leap-community/superpoint
export TT_WEIGHTS_REVISION=734450e9ffe229074f5998494ddc615475cdb20a
export TT_MESH_SHAPE=1x1 TT_DEVICE_ID=0 TT_METAL_VISIBLE_DEVICES=0

# import check with NO device (what the image's verify.sh does):
cd /tmp && $T/python_env/bin/python -c "import models.server.app as a; assert a.app"

# serve (opens the chip; hardware phase only):
cd $ROOT/models/superpoint-p150 && $T/python_env/bin/python -m uvicorn --host 0.0.0.0 --port 20000 --lifespan on models.server.app:app
# then, from another shell:
python code/models/server/smoke_test.py --url http://127.0.0.1:20000

Boot log landmarks: Loading weights ..., Opening device 0, Warming up (compiling kernels ...), Warmup complete: first forward <ms> (compile), second <ms>, then uvicorn's Application startup complete. Ctrl-C / SIGTERM closes the device.

PYTHONPATH must start with code/ so that models resolves to this repo's regular package (tt-metal's own models/ has no __init__.py and is shadowed on the host; in the image it is excluded). The device tests in code/models/tests/test_superpoint.py get their device / device_params fixtures and --device-id from the repo-local code/conftest.py (run pytest from code/; code/pytest.ini pins the rootdir). tt-metal's own conftest cannot be used next to this repo: it imports models.demos..., which the shadowing above breaks.

Package / serve / push (Blackhole host, rootless Docker)

Every command from this directory: extra_code.root: code and --out resolve against the CWD. source $ROOT/bin/docker-env.sh first (rootless Docker 28 + buildx on this box; the bare docker on PATH is podman).

ROOT=/home/deepgadget/experiments/tt-models
cd $ROOT/models/superpoint-p150
source $ROOT/bin/docker-env.sh

# offline validation (no docker, no device) -- must print VALID
$ROOT/.venv/bin/python -c "from tt_kernel.container_manifest import load_container_manifest; m = load_container_manifest('tt-model.yaml', check_sources=True); p = m.resolve_profile(); print('VALID', m.name, m.kind, p.hardware, p.mesh_device, m.weights_ref)"

# build (2.5-4 h cold; verify.sh imports the app + the verify: lines inside the image)
$ROOT/.venv/bin/tt-model package --container tt-model.yaml --out $ROOT/build

# serve + smoke (hardware phase)
$ROOT/.venv/bin/tt-model serve $ROOT/build/superpoint-p150/tt_kernel_manifest.json
python code/models/server/smoke_test.py --url http://127.0.0.1:<port serve printed>
$ROOT/.venv/bin/tt-model stop changh95/superpoint-p150

# publish (after validation only)
$ROOT/.venv/bin/tt-model push $ROOT/build/superpoint-p150 --publish

tt-cli users: tt serve changh95/superpoint-p150 / tt model stop changh95/superpoint-p150 (tt config set tools.override.tt-model $ROOT/.venv/bin/tt-model is already set here because the pinned tt-model breaks under rootless Docker).

What push does to this repo

code/ and image/ on the Hub become exactly the staged trees, so extra_code.paths lists everything under code/ worth keeping: models, kernels (fused NMS kernel source, not built into this image), sample_data, results.tsv, run_benchmark.sh. code/.gitignore is not listed and will be pruned. README.md is replaced by the generated card (all README content worth keeping lives in card.description / card.quickstart); media/, SERVING.md, .gitattributes and tt-model.yaml at the root survive. The orchestrator restores license/pipeline_tag front matter after push.

Fused path (default; `TT_FUSED=0` = legacy) -- branch `opt/superpoint-p150-megakernel`

Device-validated on the p150a 2026-09-13 (DEVICE_VALIDATION.md "Results") and made the default (code default + serve.env.TT_FUSED: "1"); TT_FUSED=0 restores the legacy path above byte-for-byte. On the fused path the lifespan opens the device with a trace region, runs the fused graph once eagerly (kernel compile), captures it into a metal trace and replays it once -- all before Application startup complete (boot log: Warming up TT_FUSED path ..., Warmup complete: compile forward ... trace capture ... traced forward ...). Per request: H2D of the [1,1,9600,32] bf16 input, one execute_trace, D2H of the row-major device NMS map (480x640) and descriptors, then threshold / border / top-k / grid_sample on the host (postprocess_from_nms_map). Requests with nms_radius != 4 read the traced softmax scores instead and run the legacy host NMS (same output, slower); response.serving_path.device_nms and /info.serving_path report which path ran. Exactness: trace, upload, device NMS and row-major outputs are bit-identical to the legacy path; the rms_norm descriptor L2-norm is bf16-rounding-level (gate: descriptor PCC >= 0.999). Host proofs: code/models/tests/test_fused_host.py; hardware plan, gates and results: DEVICE_VALIDATION.md. Measured 2026-09-13 through the container image (smoke_test.py PASS, 50 warm requests, server timing_ms medians): device_forward 5.3 ms (min 5.1), post-processing 1.3 ms, preprocess 18 ms, total 24.7 ms (~40 fps) vs the legacy path's 12.4 / 26.5 / 17.8 / 56.8 ms in the same session; keypoints and scores identical to the legacy server for nms_radius 4 (device NMS) and 3 (host fallback), descriptors within bf16 rounding (max |diff| 8.5e-4, cosine >= 0.999996). Boot log landmarks: Warming up TT_FUSED path (stages nms,rm,rms,wide, traced nms_radius 4), Warmup complete: compile forward ... trace capture ... traced forward ....

Caveats

Licence: the weights are Magic Leap "academic or non-profit organisation noncommercial research use only". Stated in the card; publishing to the public catalog inherits it.
Package name models collides with tt-metal's models/ tree. It works because the image excludes tt-metal's models/ and the port ships models/__init__.py; the filler models/common/lightweightmodule.py becomes a namespace subpackage nobody imports.
API drift: the port was validated on a tt-metal of ~v0.71 (spring 2026); this package builds against v0.78 (Aug 2026). import ttnn and every symbol the port touches (Conv2dSliceConfig, Conv2dDRAMSliceHeight, UnaryWithParam, CreateDevice, copy_host_to_device_tensor(cq_id=), max_pool2d kwargs) exist in the v0.78 tree, but conv/pool behaviour (L1 budgets of the (4, 2, 1, 1) DRAM slicing) is only proven on the chip -- re-run the PCC benchmark in the hardware phase.
Device open: ttnn.CreateDevice(device_id, l1_small_size=32768) -- the recipe of the port's untraced script (models/visualize.py); no trace region, one command queue.
Thread model: uvicorn runs the sync predict in a worker thread; the lock serialises ttnn calls, and the lifespan opened the device on the event-loop thread (same pattern as the template and other tt-dit servers).
Host venv is Python 3.10, the image is 3.12 -- the uv dry-run of the runtime packages on 3.12 resolves to torch 2.11.0+cpu, numpy 1.26.4, transformers 5.17.0, no torchvision.
transformers in the image resolves to 5.x (5.17.0 at authoring time); SuperPointForKeypointDetection exists in 4.53 .. 5.17 (>=4.53,<6 pin).