Add Tenstorrent Blackhole tt-nn port

SERVING.md

@@ -0,0 +1,80 @@
+# Serving this port with tt-model-manager
+
+This repo carries everything needed to build a **tt-model container bundle** from it, so
+`tt-model pull` / `tt-model serve` and `tt-cli` can run it. Do this on a
+**Blackhole host** (amd64 Linux + Docker >= 25).
+
+## Why not on a Mac
+
+`tt-model package --container` builds from `ghcr.io/.../ubuntu-22.04-dev-amd64` and a cold
+build is 2.5-4 hours. Packaging and pushing need **no card** (`require_host(need_devices=False)`),
+but they do need amd64. `tt-model serve` needs `/dev/tenstorrent/*`.
+
+## Steps
+
+```bash
+# 0. get tt-model (not on PyPI)
+git clone https://github.com/tenstorrent/tt-model-manager && cd tt-model-manager
+python -m venv .venv && source .venv/bin/activate && pip install -e .
+
+# 1. pull THIS repo
+hf download changh95/superpoint-blackhole --local-dir superpoint-blackhole && cd superpoint-blackhole
+
+# 2. point the manifest at your tt-metal checkout
+$EDITOR tt-model.yaml          # source.tt_metal: /path/to/tt-metal
+
+# 3. finish the serving adapter
+$EDITOR code/models/server/app.py   # see "VERIFY ON HARDWARE" markers
+
+# 4. validate with no hardware and no build
+python -c "from tt_kernel.container_manifest import ContainerManifest as M; M.load('tt-model.yaml')"
+
+# 5. build, prove, publish
+tt-model package --container tt-model.yaml
+tt-model serve   build/superpoint-blackhole/tt_kernel_manifest.json
+tt-model push    build/superpoint-blackhole --publish
+```
+
+Step 5's `--publish` implies `--public` and adds the `tt-model-catalog` tag, which is what
+lists it in the community catalog.
+
+## What the bundle push does to this repo
+
+- **`code/` and `image/` are replaced wholesale** (`_prune_removed` in `tt_kernel/hub.py`).
+  The port code staged here is exactly what `extra_code` re-stages, so it is replaced, not
+  duplicated. Files under `code/` that the allowlist does not ship get pruned.
+- **`README.md` is overwritten** by the generated card. The text you want to keep belongs in
+  `card.description` / `card.quickstart` in `tt-model.yaml` -- it is already seeded there.
+- **Root files survive**, which is why `media/` and this file live at the root.
+
+## The serving contract
+
+`kind: tt-dit-server` requires only `hardware` + `mesh_device` (no `max_num_seqs` /
+`block_size` -- those are vLLM engine settings). `runtime.app` is an ASGI target
+`"module:attr"` whose **top-level package must appear in an allowlist entry**, which is why
+`runtime.app` and `source.extra_code[0].paths` have to stay in sync. The kind installs
+fastapi / uvicorn / pydantic / pillow; `runtime.packages` adds to that set.
+
+Despite the name, this kind is not diffusion-only -- `jashansinghTT/s2-pro-blackhole` is a
+TTS model using it. It simply means "the model's own ASGI app under uvicorn".
+
+## Known sharp edges (offline-validated, not yet hardware-validated)
+
+The manifest loads clean (`load_container_manifest(..., check_sources=False)` passes, which
+includes the launcher's own `validate()`), but two things can only be settled on the box:
+
+1. **`code/` becomes the image's ONLY `models` package.** `source.code: [models/common]` is
+   staged from tt-metal into `code/models/common`, and an `extra_code` path of `models`
+   merges the port's own tree into the same `code/models/`. tt-metal's `models/` has no
+   `__init__.py`, so this relies on namespace packages resolving. Check the import in
+   `verify.sh` output on the first build.
+2. **Generically-named top-level packages.** Repos whose package is `models`, `tt`, or
+   `common` put a very common name at the root of the image's import path. If an import
+   collides, rename the package in the port repo (and update `runtime.app` +
+   `source.extra_code[].paths` together -- the launcher cross-checks them).
+
+`source.code` needs at least one tt-metal-relative entry even when all real code comes from
+`extra_code`; `models/common` is used as that minimum. If your port genuinely imports more
+from tt-metal (`locate-anything` needs `models/tt_transformers` and
+`models/demos/qwen25_vl`), list it there instead -- under-listing fails the image's own
+build-time import check on your machine, which is the cheap place to find out.

code/models/server/app.py

@@ -0,0 +1,137 @@
+# SPDX-License-Identifier: Apache-2.0
+"""ASGI serving app for SuperPoint on Tenstorrent Blackhole.
+
+Served by tt-model-manager as ``kind: tt-dit-server``::
+
+    runtime:
+      app: models.server.app:app
+
+uvicorn runs this module. The device is opened and the model built inside the ASGI
+**lifespan**, so uvicorn's ``Application startup complete`` -- which is the line
+``tt-model serve`` waits for -- means the chip is claimed and the model is warm.
+
+I/O: image -> keypoints, scores, descriptors
+"""
+from __future__ import annotations
+
+import base64
+import io
+import os
+from contextlib import asynccontextmanager
+from typing import Any
+
+import numpy as np
+import torch
+from fastapi import FastAPI, HTTPException
+from PIL import Image
+from pydantic import BaseModel, Field
+
+from models.tt.superpoint_ttnn import TtSuperPoint
+
+STATE: dict[str, Any] = {}
+
+
+def _open_device():
+    """Open the chip.
+
+    kwargs are this repo's own (from its conftest/demo), not generic defaults.
+    ``TT_MESH_SHAPE`` is set by tt-model from the manifest's ``mesh_device``; these
+    ports are single-chip, so a shape other than (1, 1) needs open_mesh_device.
+    """
+    import ttnn
+
+    shape = os.environ.get("TT_MESH_SHAPE", "(1, 1)")
+    if shape.replace(" ", "") not in ("(1,1)", "1,1"):
+        raise RuntimeError(
+            f"{shape} is a multi-chip mesh; this app opens a single device. "
+            "Use ttnn.open_mesh_device and shard the model first."
+        )
+    return ttnn.open_device(
+        device_id=int(os.environ.get("TT_DEVICE_ID", "0")),
+        l1_small_size=32768, trace_region_size=1500000000,
+    )
+
+
+def _load_reference():
+    """Load the torch reference this port wraps."""
+    from transformers import SuperPointForKeypointDetection
+    return SuperPointForKeypointDetection.from_pretrained('magic-leap-community/superpoint').eval()
+
+
+@asynccontextmanager
+async def lifespan(_app: FastAPI):
+    torch.set_grad_enabled(False)
+    device = _open_device()
+    STATE["device"] = device
+    try:
+        STATE["model"] = TtSuperPoint(torch_model=_load_reference(), device=device, input_height=480, input_width=640)
+        yield
+    finally:
+        import ttnn
+
+        STATE.pop("model", None)
+        ttnn.close_device(STATE.pop("device"))
+
+
+app = FastAPI(title="SuperPoint on Blackhole", lifespan=lifespan)
+
+
+class PredictRequest(BaseModel):
+    """One inference request. ``image`` is base64-encoded (PNG/JPEG)."""
+
+    image: str
+
+
+def _decode(b64: str) -> torch.Tensor:
+    try:
+        raw = base64.b64decode(b64, validate=True)
+        im = Image.open(io.BytesIO(raw)).convert("RGB")
+    except Exception as e:
+        raise HTTPException(status_code=400, detail=f"bad image: {e}") from None
+    arr = np.asarray(im, dtype=np.float32) / 255.0
+    return torch.from_numpy(arr).permute(2, 0, 1)[None].float()
+
+
+@app.get("/health")
+def health() -> dict:
+    return {"status": "ok" if "model" in STATE else "starting"}
+
+
+@app.get("/info")
+def info() -> dict:
+    return {
+        "model": "SuperPoint",
+        "io": "image -> keypoints, scores, descriptors",
+        "hardware": "Tenstorrent Blackhole p150a (single chip)",
+        "weights": "magic-leap-community/superpoint",
+        "source": "https://github.com/changh95/tt-superpoint",
+    }
+
+
+@app.post("/predict")
+def predict(req: PredictRequest) -> dict:
+    model = STATE.get("model")
+    if model is None:
+        raise HTTPException(status_code=503, detail="model is still starting")
+    pixel_values = _decode(req.image)
+    out = model.forward(pixel_values)
+    return {"output": _jsonable(out)}
+
+
+def _jsonable(obj: Any) -> Any:
+    """Tensors/arrays -> nested lists, so the response is plain JSON.
+
+    VERIFY ON HARDWARE: for large dense outputs (depth maps, pointmaps) returning
+    raw lists is wasteful -- prefer a PNG/NPZ response for those.
+    """
+    if isinstance(obj, torch.Tensor):
+        return obj.detach().cpu().tolist()
+    if isinstance(obj, np.ndarray):
+        return obj.tolist()
+    if isinstance(obj, dict):
+        return {k: _jsonable(v) for k, v in obj.items()}
+    if isinstance(obj, (list, tuple)):
+        return [_jsonable(v) for v in obj]
+    if hasattr(obj, "_asdict"):
+        return _jsonable(obj._asdict())
+    return obj

tt-model.yaml

@@ -0,0 +1,73 @@
+# SPDX-License-Identifier: Apache-2.0
+# tt-model-manager container manifest (schema 5.1) for SuperPoint on Blackhole.
+#
+#   tt-model package --container tt-model.yaml     # amd64 Linux + Docker>=25, ~2.5-4 h
+#   tt-model serve   build/superpoint-blackhole/tt_kernel_manifest.json   # needs a card
+#   tt-model push    build/superpoint-blackhole --publish        # adds the tt-model-catalog tag
+#
+# SET ME before building: source.tt_metal must point at YOUR built tt-metal tree.
+schema: "5.1"
+
+repo: changh95/superpoint-blackhole
+name: superpoint-blackhole
+
+# A POINTER. Weights are never baked into the image; they land in the consumer's
+# own HF cache under their own token. Pin a revision once validated.
+weights: magic-leap-community/superpoint
+
+kind: tt-dit-server
+arch: blackhole
+
+source:
+  # SET ME -- a local checkout is hermetic and lets tt-model read tt-metal's torch
+  # pin (metal_torch_pin), which verify.sh then asserts. A {repo, ref} mapping
+  # also works but skips that pin.
+  tt_metal: /path/to/tt-metal
+
+  # Relative to the tt-metal tree. At least one entry is required by the schema;
+  # this port's own code comes from extra_code below.
+  code:
+    - models/common
+
+  # This repo's code, which does NOT live in the tt-metal tree. `root: code` works
+  # because the published HF repo carries the port under code/ -- so on the box you
+  # just pull this repo and build from it.
+  extra_code:
+    - root: code
+      paths:
+        - models
+
+  ubuntu: "22.04"
+  python: "3.12"
+
+runtime:
+  app: models.server.app:app
+  mesh_shape_env: TT_MESH_SHAPE
+  # Added ON TOP of the kind's defaults (fastapi, uvicorn, pydantic>=2, pillow).
+  # torch is auto-pinned to tt-metal's pin when tt_metal is a local path.
+  packages:
+    - numpy
+    - transformers
+
+serve:
+  port: 20000
+  hardware: p150
+  mesh_device: P150
+
+# Build-time assertions, run INSIDE the finished image. No device is opened.
+verify:
+  - "import models.server.app as a; assert a.app"
+
+card:
+  description: >
+    SuperPoint on a single Tenstorrent Blackhole p150a via tt-nn.
+    image -> keypoints, scores, descriptors.
+  quickstart: |
+    ### Call it
+
+    ```bash
+    curl -s localhost:20000/health
+    curl -s localhost:20000/info
+    curl -s localhost:20000/predict -H 'Content-Type: application/json' \
+      -d "{\"image\": \"$(base64 -w0 your.png)\"}"
+    ```