SERVING.md

# Serving SuperPoint on Blackhole with tt-model-manager

This repo is the authoring source of the **tt-model container package**
`changh95/superpoint-p150` (`kind: tt-dit-server`, schema 5.1). `tt-model.yaml` is
the manifest, `code/` the port, `code/models/server/app.py` the ASGI app uvicorn runs.

| item | value |
|---|---|
| tt-metal tree | `/home/deepgadget/experiments/gbp-tt/tt-metal` (main 8b98410e730, v0.78.0-dev20260820-25, torch 2.11.0 pin) |
| weights | `magic-leap-community/superpoint` @ `734450e9ffe229074f5998494ddc615475cdb20a` (`config.json`, `model.safetensors`, `preprocessor_config.json`; 5 MB; public, ungated) |
| port source | github.com/changh95/tt-superpoint @ `e1eab66e29ff424bc9af6b1118671d9bc08e899e` (+ `code/models/server/`, `code/models/tt/postprocess.py` added here) |
| hardware | one Blackhole p150 (`hardware: p150`, `mesh_device: P150`, `TT_MESH_SHAPE=1x1`) |
| app | `models.server.app:app` |

## What the server does

Default since 2026-09-13 (`TT_FUSED` unset or `1`, also pinned in `serve.env`): pure ttnn, the
whole device graph as **one metal trace** per request with a standard-op device NMS, no custom
kernel -- see *Fused path* below for the per-request sequence and the measured numbers.
`TT_FUSED=0` restores the legacy path this section describes: pure-ttnn, **untraced**, host
NMS -- the port's `SP_TRACE_NMS=0 SP_NO_TRACE=1` configuration. Both paths: fixed 480x640
network input; every image is resized server-side. Per request (legacy):

1. base64 -> PIL RGB -> bilinear resize to 640x480 -> /255 -> fp32 `(1, 3, 480, 640)`
   (HF `SuperPointImageProcessor` defaults; the model reads channel 0 like
   `SuperPointForKeypointDetection.extract_one_channel_pixel_values`).
2. `TtSuperPoint.run_untraced(tt_in, pixel_values)`: H2D into the persistent device input,
   `run_device_compute(trace_nms=False)` (8 convs + 3 max-pools + heads, bf16/HiFi2/fp32-acc,
   softmax and descriptor L2-norm on device), `device_outputs_to_host` (D2H, **no second
   softmax**), deallocate.
3. `models.tt.postprocess.postprocess_keypoints`: fold 8x8 cells, single-pass NMS
   (`nms_radius`), threshold, border removal (4 px), top-k (`max_keypoints`), bilinear
   descriptor sampling + L2-norm. Identical to the sequence validated in
   `code/models/tests/test_superpoint.py` (score PCC 0.9971, descriptor PCC 0.9991, F1 98.8%).
4. Keypoints are scaled back to the client's pixel frame (`scale = {W/640, H/480}`).

One `threading.Lock` serialises every device call; handlers are sync and run under
`torch.inference_mode()`. Startup (weights -> device -> model -> 2 warm-up forwards on a zero
frame) happens in the ASGI lifespan, so `Application startup complete` means warm. Shutdown
deallocates the device tensors and `ttnn.close_device`s inside the 120 s SIGTERM budget.

Measured 2026-09-13 through the container image (50 warm requests, medians; legacy path):
device_forward 12.4 ms, host NMS + post 26.5 ms, total 56.8 ms. The README's 40.7 fps requires
the fused `sp_eq_mul_mask` C++ kernel (`code/kernels/`, needs a patched tt-metal) -- not what
this image runs; the fused path below gets to ~40 fps with standard ops instead.

## Environment the app reads (lifespan only, never at import)

| var | set by | meaning / default |
|---|---|---|
| `HF_MODEL` | launcher (`weights.repo`) | weights repo id; default `magic-leap-community/superpoint` |
| `TT_WEIGHTS_REVISION` | `serve.env` | commit sha passed to `from_pretrained(revision=)`; default: repo default branch |
| `SP_WEIGHTS_DIR` | you (host/offline) | local dir with `config.json` + `model.safetensors`; overrides the two above |
| `TT_MESH_SHAPE` | launcher (`runtime.mesh_shape_env`) | `1x1` (also `(1, 1)` / `1,1`); any other shape -> `RuntimeError` at startup |
| `TT_DEVICE_ID` | you | chip to open, default `0` |
| `TT_FUSED` | `serve.env` (`"1"`; the code default when unset/empty is also fused) | unset/`1` = fused serving path: ONE metal trace per request (64-byte-page input upload, encoder + heads, `rms_norm` L2-norm, standard-op device NMS at radius 4, row-major outputs), captured during warm-up before READY. `0` = the legacy untraced path above, byte-identical to the 2026-09-12 image. See `DEVICE_VALIDATION.md` |
| `TT_FUSED_STAGES` | you (device A/B only) | comma list of fused stages, default all (`wide,nms,rms,rm`); `""` = trace-only |
| `SP_TRACE_REGION` | you | `trace_region_size` bytes for `ttnn.CreateDevice` on the fused path (default 32 MiB) |
| `MESH_DEVICE` | launcher | `P150` (informational) |
| `TT_METAL_VISIBLE_DEVICES` | `serve.env` | `0` |

The launcher exports no revision, which is why `serve.env.TT_WEIGHTS_REVISION` repeats
`weights.revision`. A sha-pinned snapshot has no `refs/main`, so the app must pass the sha
(and falls back to `local_files_only=True` if the Hub is unreachable but the snapshot is cached).

## HTTP contract

| route | response |
|---|---|
| `GET /health` | `{"status": "ok" \| "starting", "model": "superpoint-p150", "device": {"arch", "id", "open"}}` (always 200) |
| `GET /info` | model/task/io, hardware, `weights {repo, revision, local_dir, loaded}`, `source {repo, commit}`, `input` (480x640, batch 1, preprocessing), `defaults`, `limits`, `serving_path` (fused: `traced=true, device_nms=true, nms_radius_traced=4, fused_stages`; legacy: `traced=false, device_nms=false`), `warmup_ms`, `descriptors` encoding, `license` |
| `GET /v1/models` | `{"object": "list", "data": [{"id": "<weights repo>", "object": "model", "owned_by": "changh95"}]}` (so OpenAI-shaped probes do not 404; not a chat API) |
| `POST /predict` | see below |

Request (`application/json`):

```json
{
  "image": "<base64 PNG/JPEG>",       // required; RGB or grayscale; any size (resized to 640x480)
  "max_keypoints": 1024,              // optional; -1 = all above threshold; cap 307200
  "keypoint_threshold": 0.005,        // optional; [0, 1]
  "nms_radius": 4,                    // optional; 0..32 canonical-frame pixels; 0 = no NMS
  "return_descriptors": true          // optional
}
```

Response:

```json
{
  "num_keypoints": N,
  "keypoints": [[x, y], ...],         // N x 2 floats, ORIGINAL image pixel coordinates
  "scores": [...],                    // N floats, always descending (keypoints/descriptors share the order)
  "original_size": {"height": H, "width": W},
  "image_size": {"height": 480, "width": 640},
  "scale": {"x": W/640, "y": H/480},  // divide keypoints by this to get network-frame coords
  "params": {"max_keypoints", "keypoint_threshold", "nms_radius", "border_removal_distance"},
  "timing_ms": {"preprocess", "device_forward", "postprocess", "total"},
  "descriptors": {                    // only when return_descriptors is true
    "format": "npz", "key": "descriptors", "dtype": "float16", "shape": [N, 256],
    "data": "<base64 NPZ>"            // numpy.load(io.BytesIO(base64.b64decode(data)))["descriptors"]
  }
}
```

Errors: `400` undecodable image or invalid field (pydantic errors are mapped to 400),
`503` while starting, `500` with `"<ExceptionType>: <message>"` on an inference failure.
One image per request.

Smoke test (the hardware phase runs it unchanged):

```bash
python code/models/server/smoke_test.py --url http://127.0.0.1:<port>
# PASS superpoint-p150: <N> keypoints on house_in_field_1080p.jpg (1600x900), ...
```

## Running on the HOST for validation (no Docker)

Uses the tree's own venv (`python_env`, Python 3.10, torch 2.11.0+cpu, transformers 5.12.1)
plus fastapi/uvicorn, which that venv lacks -- install them into a throwaway venv and append
its `site-packages` rather than touching the tree venv:

```bash
ROOT=/home/deepgadget/experiments/tt-models
T=/home/deepgadget/experiments/gbp-tt/tt-metal
export PATH=$HOME/.local/bin:$PATH
uv venv --python 3.10 /tmp/sp-http -q && uv pip install --python /tmp/sp-http/bin/python -q fastapi uvicorn

export PYTHONPATH=$ROOT/models/superpoint-p150/code:$T:$T/ttnn:$T/tools:/tmp/sp-http/lib/python3.10/site-packages
export TT_METAL_HOME=$T ARCH_NAME=blackhole
export HF_MODEL=magic-leap-community/superpoint
export TT_WEIGHTS_REVISION=734450e9ffe229074f5998494ddc615475cdb20a
export TT_MESH_SHAPE=1x1 TT_DEVICE_ID=0 TT_METAL_VISIBLE_DEVICES=0

# import check with NO device (what the image's verify.sh does):
cd /tmp && $T/python_env/bin/python -c "import models.server.app as a; assert a.app"

# serve (opens the chip; hardware phase only):
cd $ROOT/models/superpoint-p150 && $T/python_env/bin/python -m uvicorn --host 0.0.0.0 --port 20000 --lifespan on models.server.app:app
# then, from another shell:
python code/models/server/smoke_test.py --url http://127.0.0.1:20000
```

Boot log landmarks: `Loading weights ...`, `Opening device 0`, `Warming up (compiling
kernels ...)`, `Warmup complete: first forward <ms> (compile), second <ms>`, then uvicorn's
`Application startup complete`. Ctrl-C / SIGTERM closes the device.

`PYTHONPATH` must start with `code/` so that `models` resolves to this repo's regular package
(tt-metal's own `models/` has no `__init__.py` and is shadowed on the host; in the image it is
excluded). The device tests in `code/models/tests/test_superpoint.py` get their `device` /
`device_params` fixtures and `--device-id` from the repo-local `code/conftest.py` (run pytest
from `code/`; `code/pytest.ini` pins the rootdir). tt-metal's own conftest cannot be used next
to this repo: it imports `models.demos...`, which the shadowing above breaks.

## Package / serve / push (Blackhole host, rootless Docker)

Every command from **this directory**: `extra_code.root: code` and `--out` resolve against the
CWD. `source $ROOT/bin/docker-env.sh` first (rootless Docker 28 + buildx on this box; the bare
`docker` on PATH is podman).

```bash
ROOT=/home/deepgadget/experiments/tt-models
cd $ROOT/models/superpoint-p150
source $ROOT/bin/docker-env.sh

# offline validation (no docker, no device) -- must print VALID
$ROOT/.venv/bin/python -c "from tt_kernel.container_manifest import load_container_manifest; m = load_container_manifest('tt-model.yaml', check_sources=True); p = m.resolve_profile(); print('VALID', m.name, m.kind, p.hardware, p.mesh_device, m.weights_ref)"

# build (2.5-4 h cold; verify.sh imports the app + the verify: lines inside the image)
$ROOT/.venv/bin/tt-model package --container tt-model.yaml --out $ROOT/build

# serve + smoke (hardware phase)
$ROOT/.venv/bin/tt-model serve $ROOT/build/superpoint-p150/tt_kernel_manifest.json
python code/models/server/smoke_test.py --url http://127.0.0.1:<port serve printed>
$ROOT/.venv/bin/tt-model stop changh95/superpoint-p150

# publish (after validation only)
$ROOT/.venv/bin/tt-model push $ROOT/build/superpoint-p150 --publish
```

tt-cli users: `tt serve changh95/superpoint-p150` / `tt model stop changh95/superpoint-p150`
(`tt config set tools.override.tt-model $ROOT/.venv/bin/tt-model` is already set here because the
pinned tt-model breaks under rootless Docker).

## What push does to this repo

`code/` and `image/` on the Hub become exactly the staged trees, so `extra_code.paths` lists
everything under `code/` worth keeping: `models`, `kernels` (fused NMS kernel source, not built
into this image), `sample_data`, `results.tsv`, `run_benchmark.sh`. `code/.gitignore` is not
listed and will be pruned. `README.md` is replaced by the generated card (all README content
worth keeping lives in `card.description` / `card.quickstart`); `media/`, `SERVING.md`,
`.gitattributes` and `tt-model.yaml` at the root survive. The orchestrator restores
`license`/`pipeline_tag` front matter after push.

## Fused path (default; `TT_FUSED=0` = legacy) -- branch `opt/superpoint-p150-megakernel`

Device-validated on the p150a 2026-09-13 (`DEVICE_VALIDATION.md` "Results") and made the default
(code default + `serve.env.TT_FUSED: "1"`); `TT_FUSED=0` restores the legacy path above
byte-for-byte. On the fused path the lifespan opens the device with a trace region, runs the fused graph once
eagerly (kernel compile), captures it into a metal trace and replays it once -- all before
`Application startup complete` (boot log: `Warming up TT_FUSED path ...`, `Warmup complete:
compile forward ... trace capture ... traced forward ...`). Per request: H2D of the
`[1,1,9600,32]` bf16 input, one `execute_trace`, D2H of the row-major device NMS map (480x640)
and descriptors, then threshold / border / top-k / `grid_sample` on the host
(`postprocess_from_nms_map`). Requests with `nms_radius != 4` read the traced softmax scores
instead and run the legacy host NMS (same output, slower); `response.serving_path.device_nms`
and `/info.serving_path` report which path ran. Exactness: trace, upload, device NMS and
row-major outputs are bit-identical to the legacy path; the `rms_norm` descriptor L2-norm is
bf16-rounding-level (gate: descriptor PCC >= 0.999). Host proofs: `code/models/tests/test_fused_host.py`;
hardware plan, gates and results: `DEVICE_VALIDATION.md`. Measured 2026-09-13 through the
container image (`smoke_test.py` PASS, 50 warm requests, server `timing_ms` medians): device_forward
5.3 ms (min 5.1), post-processing 1.3 ms, preprocess 18 ms, total 24.7 ms (~40 fps) vs the legacy
path's 12.4 / 26.5 / 17.8 / 56.8 ms in the same session; keypoints and scores identical to the
legacy server for `nms_radius` 4 (device NMS) and 3 (host fallback), descriptors within bf16
rounding (max |diff| 8.5e-4, cosine >= 0.999996). Boot log landmarks: `Warming up TT_FUSED path
(stages nms,rm,rms,wide, traced nms_radius 4)`, `Warmup complete: compile forward ... trace capture
... traced forward ...`.

## Caveats

- **Licence**: the weights are Magic Leap "academic or non-profit organisation noncommercial
  research use only". Stated in the card; publishing to the public catalog inherits it.
- **Package name `models`** collides with tt-metal's `models/` tree. It works because the
  image excludes tt-metal's `models/` and the port ships `models/__init__.py`; the filler
  `models/common/lightweightmodule.py` becomes a namespace subpackage nobody imports.
- **API drift**: the port was validated on a tt-metal of ~v0.71 (spring 2026); this package
  builds against v0.78 (Aug 2026). `import ttnn` and every symbol the port touches
  (`Conv2dSliceConfig`, `Conv2dDRAMSliceHeight`, `UnaryWithParam`, `CreateDevice`,
  `copy_host_to_device_tensor(cq_id=)`, `max_pool2d` kwargs) exist in the v0.78 tree, but
  conv/pool behaviour (L1 budgets of the `(4, 2, 1, 1)` DRAM slicing) is only proven on the
  chip -- re-run the PCC benchmark in the hardware phase.
- **Device open**: `ttnn.CreateDevice(device_id, l1_small_size=32768)` -- the recipe of the
  port's untraced script (`models/visualize.py`); no trace region, one command queue.
- **Thread model**: uvicorn runs the sync `predict` in a worker thread; the lock serialises
  ttnn calls, and the lifespan opened the device on the event-loop thread (same pattern as the
  template and other tt-dit servers).
- **Host venv is Python 3.10**, the image is 3.12 -- the uv dry-run of the runtime packages
  on 3.12 resolves to torch 2.11.0+cpu, numpy 1.26.4, transformers 5.17.0, no torchvision.
- `transformers` in the image resolves to 5.x (5.17.0 at authoring time); `SuperPointForKeypointDetection`
  exists in 4.53 .. 5.17 (`>=4.53,<6` pin).