tt-model authoring files (fused default, megakernel pass)

SERVING.md

@@ -14,8 +14,12 @@ the manifest, `code/` the port, `code/models/server/app.py` the ASGI app uvicorn
 
 ## What the server does
 
-Pure-ttnn, **untraced**, host NMS -- the port's `SP_TRACE_NMS=0 SP_NO_TRACE=1` configuration,
-no custom kernel. Fixed 480x640 network input; every image is resized server-side. Per request:
+Default since 2026-09-13 (`TT_FUSED` unset or `1`, also pinned in `serve.env`): pure ttnn, the
+whole device graph as **one metal trace** per request with a standard-op device NMS, no custom
+kernel -- see *Fused path* below for the per-request sequence and the measured numbers.
+`TT_FUSED=0` restores the legacy path this section describes: pure-ttnn, **untraced**, host
+NMS -- the port's `SP_TRACE_NMS=0 SP_NO_TRACE=1` configuration. Both paths: fixed 480x640
+network input; every image is resized server-side. Per request (legacy):
 
 1. base64 -> PIL RGB -> bilinear resize to 640x480 -> /255 -> fp32 `(1, 3, 480, 640)`
    (HF `SuperPointImageProcessor` defaults; the model reads channel 0 like
@@ -35,9 +39,10 @@ One `threading.Lock` serialises every device call; handlers are sync and run und
 frame) happens in the ASGI lifespan, so `Application startup complete` means warm. Shutdown
 deallocates the device tensors and `ttnn.close_device`s inside the 120 s SIGTERM budget.
 
-Expected speed: ~6 fps device forward (untraced) + ~36 ms host NMS. The README's 40.7 fps
-requires trace capture and the fused `sp_eq_mul_mask` C++ kernel (`code/kernels/`, needs a
-patched tt-metal) -- deliberately not what this image runs.
+Measured 2026-09-13 through the container image (50 warm requests, medians; legacy path):
+device_forward 12.4 ms, host NMS + post 26.5 ms, total 56.8 ms. The README's 40.7 fps requires
+the fused `sp_eq_mul_mask` C++ kernel (`code/kernels/`, needs a patched tt-metal) -- not what
+this image runs; the fused path below gets to ~40 fps with standard ops instead.
 
 ## Environment the app reads (lifespan only, never at import)
 
@@ -48,6 +53,9 @@ patched tt-metal) -- deliberately not what this image runs.
 | `SP_WEIGHTS_DIR` | you (host/offline) | local dir with `config.json` + `model.safetensors`; overrides the two above |
 | `TT_MESH_SHAPE` | launcher (`runtime.mesh_shape_env`) | `1x1` (also `(1, 1)` / `1,1`); any other shape -> `RuntimeError` at startup |
 | `TT_DEVICE_ID` | you | chip to open, default `0` |
+| `TT_FUSED` | `serve.env` (`"1"`; the code default when unset/empty is also fused) | unset/`1` = fused serving path: ONE metal trace per request (64-byte-page input upload, encoder + heads, `rms_norm` L2-norm, standard-op device NMS at radius 4, row-major outputs), captured during warm-up before READY. `0` = the legacy untraced path above, byte-identical to the 2026-09-12 image. See `DEVICE_VALIDATION.md` |
+| `TT_FUSED_STAGES` | you (device A/B only) | comma list of fused stages, default all (`wide,nms,rms,rm`); `""` = trace-only |
+| `SP_TRACE_REGION` | you | `trace_region_size` bytes for `ttnn.CreateDevice` on the fused path (default 32 MiB) |
 | `MESH_DEVICE` | launcher | `P150` (informational) |
 | `TT_METAL_VISIBLE_DEVICES` | `serve.env` | `0` |
 
@@ -60,7 +68,7 @@ The launcher exports no revision, which is why `serve.env.TT_WEIGHTS_REVISION` r
 | route | response |
 |---|---|
 | `GET /health` | `{"status": "ok" \| "starting", "model": "superpoint-p150", "device": {"arch", "id", "open"}}` (always 200) |
-| `GET /info` | model/task/io, hardware, `weights {repo, revision, local_dir, loaded}`, `source {repo, commit}`, `input` (480x640, batch 1, preprocessing), `defaults`, `limits`, `serving_path` (traced=false, device_nms=false), `warmup_ms`, `descriptors` encoding, `license` |
+| `GET /info` | model/task/io, hardware, `weights {repo, revision, local_dir, loaded}`, `source {repo, commit}`, `input` (480x640, batch 1, preprocessing), `defaults`, `limits`, `serving_path` (fused: `traced=true, device_nms=true, nms_radius_traced=4, fused_stages`; legacy: `traced=false, device_nms=false`), `warmup_ms`, `descriptors` encoding, `license` |
 | `GET /v1/models` | `{"object": "list", "data": [{"id": "<weights repo>", "object": "model", "owned_by": "changh95"}]}` (so OpenAI-shaped probes do not 404; not a chat API) |
 | `POST /predict` | see below |
 
@@ -139,8 +147,10 @@ kernels ...)`, `Warmup complete: first forward <ms> (compile), second <ms>`, the
 
 `PYTHONPATH` must start with `code/` so that `models` resolves to this repo's regular package
 (tt-metal's own `models/` has no `__init__.py` and is shadowed on the host; in the image it is
-excluded). The benchmark `code/models/tests/test_superpoint.py` still needs a tt-metal checkout as
-pytest rootdir for its `device` fixture -- it is unchanged by the packaging work.
+excluded). The device tests in `code/models/tests/test_superpoint.py` get their `device` /
+`device_params` fixtures and `--device-id` from the repo-local `code/conftest.py` (run pytest
+from `code/`; `code/pytest.ini` pins the rootdir). tt-metal's own conftest cannot be used next
+to this repo: it imports `models.demos...`, which the shadowing above breaks.
 
 ## Package / serve / push (Blackhole host, rootless Docker)
 
@@ -182,6 +192,30 @@ worth keeping lives in `card.description` / `card.quickstart`); `media/`, `SERVI
 `.gitattributes` and `tt-model.yaml` at the root survive. The orchestrator restores
 `license`/`pipeline_tag` front matter after push.
 
+## Fused path (default; `TT_FUSED=0` = legacy) -- branch `opt/superpoint-p150-megakernel`
+
+Device-validated on the p150a 2026-09-13 (`DEVICE_VALIDATION.md` "Results") and made the default
+(code default + `serve.env.TT_FUSED: "1"`); `TT_FUSED=0` restores the legacy path above
+byte-for-byte. On the fused path the lifespan opens the device with a trace region, runs the fused graph once
+eagerly (kernel compile), captures it into a metal trace and replays it once -- all before
+`Application startup complete` (boot log: `Warming up TT_FUSED path ...`, `Warmup complete:
+compile forward ... trace capture ... traced forward ...`). Per request: H2D of the
+`[1,1,9600,32]` bf16 input, one `execute_trace`, D2H of the row-major device NMS map (480x640)
+and descriptors, then threshold / border / top-k / `grid_sample` on the host
+(`postprocess_from_nms_map`). Requests with `nms_radius != 4` read the traced softmax scores
+instead and run the legacy host NMS (same output, slower); `response.serving_path.device_nms`
+and `/info.serving_path` report which path ran. Exactness: trace, upload, device NMS and
+row-major outputs are bit-identical to the legacy path; the `rms_norm` descriptor L2-norm is
+bf16-rounding-level (gate: descriptor PCC >= 0.999). Host proofs: `code/models/tests/test_fused_host.py`;
+hardware plan, gates and results: `DEVICE_VALIDATION.md`. Measured 2026-09-13 through the
+container image (`smoke_test.py` PASS, 50 warm requests, server `timing_ms` medians): device_forward
+5.3 ms (min 5.1), post-processing 1.3 ms, preprocess 18 ms, total 24.7 ms (~40 fps) vs the legacy
+path's 12.4 / 26.5 / 17.8 / 56.8 ms in the same session; keypoints and scores identical to the
+legacy server for `nms_radius` 4 (device NMS) and 3 (host fallback), descriptors within bf16
+rounding (max |diff| 8.5e-4, cosine >= 0.999996). Boot log landmarks: `Warming up TT_FUSED path
+(stages nms,rm,rms,wide, traced nms_radius 4)`, `Warmup complete: compile forward ... trace capture
+... traced forward ...`.
+
 ## Caveats
 
 - **Licence**: the weights are Magic Leap "academic or non-profit organisation noncommercial

tt-model.yaml

@@ -60,6 +60,10 @@ serve:
   env:
     TT_WEIGHTS_REVISION: "734450e9ffe229074f5998494ddc615475cdb20a"
     TT_METAL_VISIBLE_DEVICES: "0"
+    # Fused serving path (one metal trace per request, standard-op device NMS), device-validated
+    # 2026-09-13 (DEVICE_VALIDATION.md "Results"); it is also the code default. "0" = the legacy
+    # untraced host-NMS path of the 2026-09-12 image.
+    TT_FUSED: "1"
 
 # Build-time assertions run INSIDE the finished image as uid 1000, no device, no weights.
 verify:
@@ -99,7 +103,8 @@ card:
      "scores": [0.609375, 0.589844, 0.582031],
      "original_size": {"height": 900, "width": 1600}, "image_size": {"height": 480, "width": 640}, "scale": {"x": 2.5, "y": 1.875},
      "descriptors": {"format": "npz", "key": "descriptors", "dtype": "float16", "shape": [539, 256], "data": "..."},
-     "timing_ms": {"preprocess": 19.4, "device_forward": 12.5, "postprocess": 32.5, "total": 64.4}}
+     "serving_path": {"traced": true, "device_nms": true},
+     "timing_ms": {"preprocess": 17.7, "device_forward": 5.3, "postprocess": 1.3, "total": 24.3}}
     ```
 
     - `keypoints` are `[x, y]` in original image pixels, sorted by descending `scores`; the network frame is 480×640 and `scale` = original / network.
@@ -116,16 +121,18 @@ card:
     | Metric | Value |
     |---|---:|
     | Pre-NMS score map · descriptor map PCC vs fp32 torch reference | 0.9971 · 0.9991 |
-    | Keypoint set vs reference (natural image, top-500, 2 px) | recall 98.80% · precision 98.80% · F1 98.80% |
-    | Inference, served over HTTP (warm, batch 1, 480×640, 1600×900 JPEG in) | ~12–20 ms device · ~65 ms end-to-end (~15 FPS) |
+    | Keypoint set vs reference (natural image, top-500, 2 px) | recall 98.20% · precision 99.40% · F1 98.80% |
+    | Inference, served over HTTP (warm, batch 1, 480×640, 1600×900 JPEG in; median of 50 requests) | 5.3 ms device (trace + device NMS) · 1.3 ms host post-processing · 18 ms JPEG decode/resize · 24.7 ms end-to-end (~40 FPS) |
+    | Same, legacy path (`TT_FUSED=0`: untraced, host NMS) | 12.4 ms device · 26.5 ms host NMS · 56.8 ms end-to-end (~18 FPS) |
 
     ### Caveats
 
     - Every image is resized to 480×640 (bilinear, /255, channel 0); one image per request, batch 1, requests serialised on the chip.
-    - This server runs the untraced pure-ttnn path with host single-pass NMS (bf16 + HiFi2 + fp32 accumulate; bfloat8/LoFi drop score PCC to ~0.91). The port README's 40.7 fps needs `ttnn.trace` plus the fused `sp_eq_mul_mask` C++ kernel in `code/kernels/`, which is not built into this image.
+    - This server runs the whole device graph as one metal trace per request with a standard-op device NMS (radius 4; bit-identical to the host single-pass NMS) and an `rms_norm` descriptor L2-norm (bf16-rounding-level vs the legacy chain; PCC 0.9991 either way). bf16 + HiFi2 + fp32 accumulate throughout (bfloat8/LoFi drop score PCC to ~0.91). No custom kernel: the port README's `sp_eq_mul_mask` path is not built into this image.
+    - `nms_radius` other than 4 falls back to the host NMS on the traced scores (same keypoints, ~25 ms slower); `TT_FUSED=0` in the environment restores the untraced legacy path (validated 2026-09-12).
     - Weights are research-only: the Magic Leap SuperPoint licence allows academic or non-profit organisation NONCOMMERCIAL research use.
     - Not an OpenAI-compatible API; `GET /v1/models` is a stub so the tt-model ready card does not 404.
-    - Validated on tt-metal `v0.78.0-dev20260820` (main `8b98410e730`), single p150a only.
+    - Validated on tt-metal `v0.78.0-dev20260820` (main `8b98410e730`), single p150a only; numbers above measured 2026-09-13 through this container image (`DEVICE_VALIDATION.md`).
 
     ### Licensing