GPU_COMPARISON.md: same-host RTX 5090 benchmark of the torch reference

GPU_COMPARISON.md

@@ -0,0 +1,161 @@
+# superpoint-p150 — Blackhole p150a vs RTX 5090 (same host, same weights, same input)
+
+Date 2026-09-14. Facts only; every number below is measured in this pass on the GPU or copied
+(with its source line) from the p150a validation reports. The p150a was NOT touched.
+
+## What was run
+
+| | |
+|---|---|
+| Model | SuperPoint, the port's own torch reference: `transformers.SuperPointForKeypointDetection` (`models/superpoint-p150/code/models/reference/superpoint_reference.py::load_reference_model`, the same class `server/app.py::_load_reference` wraps into `TtSuperPoint`), 1,300,865 parameters, transformers 5.12.1 |
+| Weights | `magic-leap-community/superpoint` @ `734450e9ffe229074f5998494ddc615475cdb20a` (tt-model.yaml `weights.revision` = `serve.env.TT_WEIGHTS_REVISION`), `model.safetensors` from the HF cache `~/.cache/huggingface/hub/models--magic-leap-community--superpoint/snapshots/734450e9…`, `HF_HUB_OFFLINE=1` |
+| Input | `code/sample_data/house_in_field_1080p.jpg` (1600x900 RGB JPEG — the frame the p150a served numbers were taken on, 539 kp; `media/sample.png` is its 480x640 keypoint visualisation, not an input) -> `server/app.py::_preprocess` copied verbatim: PIL bilinear resize to 640x480, /255 -> `pixel_values [1,3,480,640]` fp32, batch 1; the model reads channel 0 (R) (`extract_one_channel_pixel_values`) |
+| GPU | NVIDIA GeForce RTX 5090 (sm_120), driver 580.126.18, power limit 600 W, 32607 MiB; idle 29.8 W |
+| venv | `/home/deepgadget/experiments/tt-models/.venv-gpu/main` — Python 3.12.13, torch 2.11.0+cu128, CUDA 12.8, cuDNN 9.19, torchvision 0.26.0, transformers 5.12.1, numpy 1.26.4, safetensors 0.8.0, huggingface_hub 1.31.0, pillow 12.3.0, triton 3.6.0 |
+| Repo state | `models/superpoint-p150` @ `8528a70` (branch `tt-model-package`), read-only; the port's torch-only `models/tt/postprocess.py` (fold_scores / simple_nms / postprocess_from_nms_map) is imported unchanged |
+| Script | `logs/gpu-vs-p150/superpoint/bench_superpoint_gpu.py` (uses `logs/gpu-vs-p150/bench_common.py`); logs `full_run.log` (eager + served-like), `compile_only_nocache_{tf32,bf16_autocast,fp16_autocast}.log` (the `torch.compile` rows), `probe_compile_*.{py,log}` (the inductor-cache investigation below); raw JSON `reports/gpu-vs-p150/superpoint.json` (= `logs/gpu-vs-p150/superpoint/result.json`); CPU reference tensors `cpu_fp32_reference.pt` |
+| Commands | `HF_HUB_OFFLINE=1 .venv-gpu/main/bin/python bench_superpoint_gpu.py --iters 50 --warmup 10` then, one process per precision, `TORCHINDUCTOR_FORCE_DISABLE_CACHES=1 HF_HUB_OFFLINE=1 … bench_superpoint_gpu.py --iters 50 --warmup 10 --compile-only --compile-precisions <tf32|bf16_autocast|fp16_autocast>` |
+| Loop | per precision 10 warm-ups + 50 timed iterations, `torch.cuda.synchronize()` before/after each; wall-clock (perf_counter) is the primary number, CUDA-event time recorded alongside (within 0.01 ms of wall) |
+| p150a source | `reports/gpu-vs-p150/p150_numbers.json` -> `reports/megakernel/PUBLISH_SUMMARY.md:11` (Hub `tt serve`: device_forward 5.11 ms, total 23.94 ms), `models/superpoint-p150/DEVICE_VALIDATION.md` "## Results" (served table: fused 5.28 / 5.07 ms device_forward, preprocess 17.5-18.1 ms, postprocess 0.96-1.29 ms, total 23.79-24.71 ms; fused A/B `fused_h2d_plus_trace_ms` 3.99), `reports/megakernel/VALIDATION_SUMMARY.md:32` |
+
+GPU forward variants (all the same weights, all on the GPU):
+
+- **device_fwd** (the primary number) = encoder + keypoint head (65-ch softmax) + descriptor head (L2-normalised 256x60x80) + the port's `fold_scores` (drop dustbin, 8x8 unfold) + single-pass NMS radius 4 (`simple_nms`, one max-pool + eq) -> `nms_map [1,480,640]` + `descriptors [1,256,60,80]`. This is exactly the work inside the p150a `timing_ms.device_forward` key (whole-graph metal trace incl. on-device NMS-T, readback of the NMS map and the descriptor map). The host post-processing after it (threshold 0.005 / border 4 / top-1024 / grid_sample) is the same port code on both sides.
+- **dense** = encoder + heads only (no fold/NMS): the tensors the port's PCC gate compares (informational).
+- **hf_full** = `SuperPointForKeypointDetection.forward` as shipped by transformers: dense + HF 3-pass NMS + threshold/border/top-k (`max_keypoints=-1` in the config -> all) + `grid_sample`, on the GPU (informational; data-dependent shapes force a sync inside).
+
+Timing definitions (they match the p150a `timing_ms` keys):
+
+- **incl_h2d** = `pixel_values.to("cuda")` (3.7 MB, pageable host tensor as produced by the server preprocess) + device_fwd + `nms_map.cpu()` + `descriptors.cpu()` (1.2 + 4.9 MB). Compare with p150a `timing_ms.device_forward` = upload of the 480x640 frame + trace replay + readback of both maps (**5.11 ms**, PUBLISH_SUMMARY.md:11; 5.28 / 5.07 in the DEVICE_VALIDATION sessions).
+- **excl_h2d** = device_fwd only, input already resident, outputs left on the device.
+- **served-like** = base64 decode + JPEG decode (PIL, convert RGB) + `_preprocess` + incl_h2d device_fwd + `postprocess_from_nms_map` + `argsort` (the interval `server/app.py` reports as `timing_ms.total`; response JSON / npz encode is outside that key on both sides). Compare with p150a `timing_ms.total` (**23.94 ms**, PUBLISH_SUMMARY.md:11).
+
+## Correctness check (GPU vs CPU fp32 reference)
+
+CPU fp32 (same process, same `pixel_values`): dense 119 ms, device_fwd 150 ms, HF full forward 208 ms; served-path output 526 keypoints
+(top-3 in original pixels `[610.0, 703.125] [1042.5, 446.25] [1122.5, 442.5]`, scores 0.6283 / 0.6044 / 0.5985 — the same three positions the
+published p150a card lists, at bf16-rounded scores 0.6094 / 0.5898 / 0.5820). The p150a served 539 kp on this frame: the 13 extra points come from bf16
+score rounding around the 0.005 threshold / NMS ties (its pre-NMS score PCC vs this reference is 0.9971, VS:32).
+
+| GPU precision | pre-NMS score map PCC (p150a gate metric) | descriptor map PCC (p150a gate metric) | NMS map PCC | kp top-500 @2 px recall / precision / F1 | served output vs CPU (526 kp) |
+|---|---:|---:|---:|---:|---|
+| **fp32 strict** | **1.000000** (max abs diff 2.7e-6) | **1.000000** (1.1e-6) | 1.000000 | 0.998 / 0.998 / 0.998 | 526 kp, **identical keypoint set**, scores within 1.9e-6, descriptor cosine >= 0.9999993 |
+| tf32 | 0.999994 (2.8e-3) | 0.999998 (1.4e-3) | 0.999937 | 1.000 / 1.000 / 1.000 | 526 kp; 99.62 % of ref keypoints at the exact pixel, 99.81 % within 1 px |
+| bf16 autocast | 0.999155 (3.1e-2) | 0.999484 (1.9e-2) | 0.983589 | 0.992 / 0.994 / 0.993 | 530 kp; 95.8 % exact, 99.6 % within 1 px |
+| fp16 autocast | 0.999963 (8.4e-3) | 0.999982 (5.4e-3) | 0.998310 | 0.996 / 1.000 / 0.998 | 527 kp; 99.0 % exact, 99.6 % within 1 px |
+| p150a (bf16 device, VS:32 / DEVICE_VALIDATION Results) | 0.997109 | 0.999085 | n/a (NMS map bit-identical to the host fold+NMS of its own scores) | F1 0.98796 | 539 kp |
+
+PCC > 0.999 holds for fp32 strict (1.000000 on both gate tensors; the served keypoint set is identical to the CPU one), so the GPU runs the right
+model. The top-500 F1 of 0.998 rather than 1.0 in fp32 comes from one ~1e-6 score tie at rank 500 flipping the k-th point — the full served set is identical.
+fp16 autocast is numerically fine (0.99996 / 0.99998, above the port's 0.997 / 0.999 gates); bf16 autocast (the p150a's own activation class) is
+at 0.99916 / 0.99948 — above the p150a's own 0.9971 / 0.99909, with a slightly lower keypoint F1 (0.993 vs 0.988 for the p150a).
+
+## GPU latency (batch 1, 480x640, median / min / p90 of 50 iterations, wall-clock ms)
+
+Eager PyTorch, **device_fwd** (== p150a device_forward work):
+
+| precision | incl_h2d median / min / p90 | excl_h2d median / min / p90 | CUDA-event excl | first call ms | power mean W (excl loop) | power mean W (incl loop) | GPU util % (excl) | peak mem alloc / reserved MiB |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|
+| fp32 strict (`allow_tf32=False`, `'highest'`) | **2.938** / 2.908 / 2.964 | **2.259** / 2.238 / 3.516 | 2.247 | 2.3 (199.9 on the very first call incl. CUDA/cuDNN init) | 548.0 | 299.8 | 95.5 | 176 / 404 |
+| tf32 (`allow_tf32=True`, `'high'`; PyTorch default is `'highest'`) | **2.097** / 2.072 / 2.118 | **1.425** / 1.416 / 1.434 | 1.415 | 1.5 | 409.5 | 423.7 | 88.7 | 327 / 406 |
+| bf16 autocast (+TF32 remainder) | **1.586** / 1.559 / 1.616 | **0.913** / 0.901 / 0.921 | 0.905 | 0.9 | 397.5 | 346.3 | 85.4 | 175 / 406 |
+| fp16 autocast (+TF32 remainder) | **1.526** / 1.502 / 1.555 | **0.857** / 0.847 / 0.864 | 0.848 | 0.9 | 396.3 | 335.8 | 84.2 | 175 / 406 |
+| tf32, pinned host input (informational) | 2.046 / 2.023 / 2.065 | | | | | | | |
+| bf16 autocast, pinned host input (informational) | 1.546 / 1.520 / 1.573 | | | | | | | |
+
+The incl-excl gap of ~0.67 ms is the PCIe traffic (3.7 MB pageable upload + 6.1 MB readback of the two fp32 maps); the fp32-strict excl loop had a
+few slow iterations (p90 3.52, max 4.66 ms) while its median/min are tight. The other two variants (same precisions, incl_h2d / excl_h2d medians):
+
+| precision | dense (no fold/NMS) | hf_full (transformers forward incl. 3-pass NMS + top-k + grid_sample, kp count) |
+|---|---:|---:|
+| fp32 strict | 2.911 / 2.230 | 2.973 / 2.674 (581 kp) |
+| tf32 | 2.072 / 1.397 | 2.129 / 1.833 (581 kp) |
+| bf16 autocast | 1.563 / 0.892 | 1.654 / 1.350 (587 kp) |
+| fp16 autocast | 1.512 / 0.837 | 1.588 / 1.296 (582 kp) |
+
+The port's fold + single-pass NMS costs ~0.03 ms on the GPU; the HF 3-pass NMS + host-style keypoint extraction adds ~0.45 ms and returns every
+point above threshold (581, no top-k, `max_keypoints=-1`).
+
+`torch.compile` (inductor, `dynamic=False`, device_fwd; each precision compiled in its own fresh process with `TORCHINDUCTOR_FORCE_DISABLE_CACHES=1`
+— see the note below; compile time 2-3.4 s, far under the 5-min budget):
+
+| variant | compile s | incl_h2d median / min / p90 | excl_h2d median / min / p90 | power W (excl loop) | peak mem MiB | NMS map PCC vs CPU fp32 | kp F1 top-500 | served kp (exact-pixel match to CPU) |
+|---|---:|---:|---:|---:|---:|---:|---:|---|
+| tf32 + compile default | 3.4 | 1.832 / 1.802 / 1.863 | 1.134 / 1.127 / 1.142 | 395.4 | 251 | 0.999937 | 1.000 | 526 (99.6 %) |
+| tf32 + compile reduce-overhead (CUDA graphs) | 1.8 | **1.776** / 1.750 / 1.801 | **1.051** / 1.045 / 1.058 | 421.0 | 18 | 0.999937 | 1.000 | 526 (99.6 %) |
+| bf16 autocast + compile default | 3.4 | 1.359 / 1.320 / 1.387 | 0.647 / 0.636 / 0.659 | 362.1 | 138 | 0.979078 | 0.994 | 527 (95.1 %) |
+| bf16 autocast + compile reduce-overhead | 1.8 | **1.291** / 1.269 / 1.326 | **0.588** / 0.581 / 0.600 | 381.3 | 18 | 0.979078 | 0.994 | 527 (95.1 %) |
+| fp16 autocast + compile default | 3.3 | 1.286 / 1.256 / 1.328 | 0.588 / 0.580 / 0.596 | 361.6 | 100 | 0.996632 | 0.998 | 526 (98.9 %) |
+| fp16 autocast + compile reduce-overhead | 1.9 | **1.211** / 1.187 / 1.238 | **0.505** / 0.497 / 0.514 | 388.7 | 18 | 0.996632 | 0.998 | 526 (98.9 %) |
+
+Compiled descriptor-map PCC: tf32 0.999998, bf16 0.999678, fp16 0.999987 (pre-NMS score map is not an output of the compiled graph; its eager value applies).
+Compile note (`probe_compile_single.py`, `probe_compile_graph.py`): with inductor's default on-disk cache the fp16 compile returned the *bf16*
+kernels (bit-identical outputs and latency to the bf16 compile in fresh processes, "compile" 0.1 s) although the dynamo/AOT graph carried
+`_to_copy(dtype=float16)` casts — a cache-key collision across autocast dtypes in torch 2.11.0+cu128. The rows above were therefore re-measured
+with `TORCHINDUCTOR_FORCE_DISABLE_CACHES=1`, one precision per process; with caches off the compiled fp16 output tracks eager fp16 (NMS-map PCC 0.998
+vs eager) and differs from bf16. The bf16 rows were identical with and without the cache.
+
+Other facts: model to cuda 0.002 s (5 MB of weights; CPU load from safetensors 0.02 s); first fp32 call 200 ms (CUDA/cuDNN init); idle GPU power
+29.8 W; GPU utilisation 84-96 % in the eager excl loops (this graph is 12 small convs — kernel-launch-bound at batch 1, which is why CUDA graphs and
+autocast both help and why fp32 strict draws the most power).
+
+Served-like loop (same host work as `server/app.py::predict`, 50 iterations after 10 warm-ups, medians ms):
+
+| GPU precision | decode (base64 + JPEG 1600x900) | preprocess (PIL resize + /255) | device_forward (incl_h2d) | postprocess | **total** | p90 total | power W (loop) | keypoints |
+|---|---:|---:|---:|---:|---:|---:|---:|---|
+| fp32 strict | 11.74 | 6.18 | 3.00 | 1.52 | **22.75** | 24.33 | 235.2 | 526 |
+| tf32 | 11.93 | 6.03 | 2.15 | 2.21 | **22.21** | 23.15 | 103.5 | 526 |
+| bf16 autocast | 11.90 | 5.96 | 1.64 | 2.26 | **21.45** | 22.13 | 93.6 | 530 |
+| fp16 autocast | 12.05 | 6.15 | 1.59 | 2.19 | **21.90** | 23.36 | 91.0 | 527 |
+| p150a (PUBLISH_SUMMARY.md:11; split DEVICE_VALIDATION.md Results, Hub run) | ~17.5-18.1 (decode + resize as one key) | | 5.11 | 0.96-1.29 | **23.94** | | not measured | 539 |
+
+The host stages dominate on both sides: the same PIL JPEG decode + resize takes ~18 ms here (11.9 + 6.0; the p150a container reports 17.5-18.1 ms for the
+same two steps as one key), so the served totals differ by roughly the inference term only. The GPU postprocess (torch CPU ops in this venv) is
+1.5-2.3 ms vs 0.96-1.29 ms in the p150a container — same code, different host process / torch build; not a device difference.
+
+## Comparison with the p150a (matching definitions)
+
+Ratio = p150a ms / GPU ms (> 1 means the GPU is faster). p150a precision: bf16 activations, HiFi2 + fp32 accumulate on device, one metal trace with
+device NMS-T (DEVICE_VALIDATION.md Results). GPU precision per row as stated.
+
+| row | p150a (definition) | GPU precision | GPU ms | ratio p150a/GPU |
+|---|---:|---|---:|---:|
+| device forward (p150a `timing_ms.device_forward` = upload + trace incl. NMS + readback vs GPU device_fwd incl_h2d) | 5.11 (PUBLISH_SUMMARY.md:11, Hub tt serve; 5.28 / 5.07 DEVICE_VALIDATION) | fp32 strict | 2.938 | **1.74** |
+| | | tf32 | 2.097 | **2.44** |
+| | | bf16 autocast (the p150a's precision class) | 1.586 | **3.22** |
+| | | fp16 autocast | 1.526 | **3.35** |
+| | | tf32 + compile reduce-overhead | 1.776 | **2.88** |
+| | | bf16 autocast + compile reduce-overhead | 1.291 | **3.96** |
+| | | fp16 autocast + compile reduce-overhead | 1.211 | **4.22** |
+| GPU forward only (excl_h2d, no PCIe) vs the same p150a 5.11 (which cannot exclude its transfers) | 5.11 | fp32 strict / tf32 / bf16 / fp16 | 2.259 / 1.425 / 0.913 / 0.857 | 2.26 / 3.59 / 5.60 / 5.96 |
+| | | tf32 / bf16 / fp16 + compile reduce-overhead | 1.051 / 0.588 / 0.505 | 4.86 / 8.69 / 10.12 |
+| GPU forward only (excl_h2d) vs p150a `fused_h2d_plus_trace_ms` 3.99 (DEVICE_VALIDATION Results, host harness: upload + trace, no readback/convert) | 3.99 | fp32 strict / tf32 / bf16 / fp16 | 2.259 / 1.425 / 0.913 / 0.857 | 1.77 / 2.80 / 4.37 / 4.66 |
+| served e2e (p150a `timing_ms.total` vs GPU served-like total) | 23.94 (PUBLISH_SUMMARY.md:11) | fp32 strict | 22.75 | **1.05** |
+| | | tf32 | 22.21 | **1.08** |
+| | | bf16 autocast | 21.45 | **1.12** |
+| | | fp16 autocast | 21.90 | **1.09** |
+
+Reading: on the device-forward definition (upload + network + NMS + readback) the eager RTX 5090 is 1.7x faster than the p150a's fused trace in strict
+fp32 (2.94 vs 5.11 ms) and 3.2x faster in the p150a's own precision class (bf16 autocast, 1.59 ms); the best GPU configuration measured (fp16 autocast +
+inductor + CUDA graphs) is 4.2x faster (1.21 ms) with better agreement to the fp32 reference than the p150a (kp F1 0.998 vs 0.988). About 0.7 ms of every
+GPU incl_h2d number is PCIe traffic for the 6 MB of fp32 maps — the fold/NMS itself is ~0.03 ms. End-to-end the ~18 ms of identical host JPEG decode /
+resize work on both sides compresses the gap to 1.05-1.12x (22-23 vs 23.94 ms): for this model the served latency is host-bound on either accelerator.
+
+Not measured / not claimed: p150a power (not measured in any pass -> no power or efficiency comparison; the GPU drew 360-550 W mean during the dense
+excl loops, 90-235 W in the host-bound served-like loops, 30 W idle). p150a numbers were not re-measured. The GPU numbers exclude HTTP/JSON framing
+and the npz/base64 response encode, as do the p150a `timing_ms` keys. Keypoint counts differ across precisions (526-530 here, 539 on the p150a)
+because points near the 0.005 threshold / NMS ties flip with rounding; the top-ranked keypoints and their positions agree.
+
+## Reproduce
+
+```bash
+cd /home/deepgadget/experiments/tt-models/logs/gpu-vs-p150/superpoint
+HF_HUB_OFFLINE=1 /home/deepgadget/experiments/tt-models/.venv-gpu/main/bin/python bench_superpoint_gpu.py --iters 50 --warmup 10 | tee full_run.log
+for P in tf32 bf16_autocast fp16_autocast; do   # one autocast dtype per process, inductor caches off (see the compile note)
+  TORCHINDUCTOR_FORCE_DISABLE_CACHES=1 HF_HUB_OFFLINE=1 /home/deepgadget/experiments/tt-models/.venv-gpu/main/bin/python \
+    bench_superpoint_gpu.py --iters 50 --warmup 10 --compile-only --compile-precisions $P | tee compile_only_nocache_$P.log
+done
+# outputs: /home/deepgadget/experiments/tt-models/reports/gpu-vs-p150/superpoint.json, ./result.json, ./cpu_fp32_reference.pt
+```
+
+GPU released after the run: `nvidia-smi --query-compute-apps=pid --format=csv,noheader` -> empty.