OPT_REPORT.md

mast3r-p150 — 12x10 optimisation report (Galaxy BH chips 9, 3, 18)

Status (2026-10-05): p150 ETH-dispatch compliance done (section "p150 ETH-dispatch compliance 2026-10-05"): the server, the harnesses and the Python API default to ETH dispatch + 1 CQ + 12x10; outputs bit-identical to the old Tensix 2-CQ served default; served forward 25.6 -> 26.8 ms (lost CQ1 readback overlap), all gates pass.

Previous status (2026-10-04): audit integration done (section "Audit integration 2026-10-04"): served npz request -72 % server time / -60 % client wall (stored npz), png -34 %, new binary route /predict_npz (-70 % client wall), and LoFi-RNE MLP linears (served e2e -7.2 %, all gates pass, two ungated metrics slightly worse, disclosed).**

Previous status: stopped: round 9 finished (this workflow's optimisation round 2); the backlog below is still open.** Round 9 kept three small bit-identical plumbing steps (esplit, demb, hrqk): traced device span 23.531 -> 23.487 ms (-0.044 ms, -0.19 %, chip 9, same session). That is below the host-wall noise; served u8 e2e is unchanged within noise (27.7-28.4 ms). The two planned big items were measured first and turned out to be capped: the 1.3-2.2 ms "host / dispatch gap" is the chip's AICLK throttling under load (not host or dispatch), and a custom SDPA compute kernel alone can save at most ~10 us per encoder call. All gates pass with identical numbers (every kept step is bit-identical on the eth12 float and the served u8 paths); the server smoke test passed 4 times out of 4. No chip faulted.

Model: DUSt3R ViT-L/16 + dual-branch decoder + 2 DPT heads (naver/DUSt3R_ViTLarge_BaseDecoder_512_dpt @ 61c57447), one 512x512 image pair per request, batch 1. tt-metal 8b98410e730 + ETH-dispatch patch (unmodified; every new kernel of round 3 is model-local and runs through ttnn.generic_op). Grid: 12x10 (TT_METAL_CORE_GRID_OVERRIDE_TODEPRECATE=11,9), ETH dispatch, 1 CQ (= p150 + ETH dispatch). Baseline and its profile: OPT_BASELINE.md. All knobs are MAST3R_OPT entries in code/models/demos/mast3r/tt/fused.py (all = default and pinned in tt-model.yaml; none = the pre-optimisation fused graph). Default set now: out, mm, dpt, l1, dptf, gelut, lnfold, phase, hostcol, phase0, decb, resl1, ropes, sym, tsplit, sdpa2, hrope, hcat, mln, mc2d, mc2dr, resl1d, addln, cdbl, pil, til, mm32, dmm, dadd, addln2, rchain, dlin, dfront, dshard, ups1, tailf, pemm, pcat, p1hs, hf3, gpoly, qkx, sdbl, pilhs, tups, sgat, tapm, obs, obsd, resmm, resmmd, esplit, demb, hrqk, lofie, lofid (gelua and lofi are off; lofie / lofid since the 2026-10-04 audit integration, below). Since 2026-10-05 the serve config pins MAST3R_DISPATCH=auto + MAST3R_CQS=1 (ETH dispatch, 1 CQ, 12x10 = the p150 configuration; section "p150 ETH-dispatch compliance 2026-10-05"). Before that it pinned MAST3R_CQS=2 (split trace, head-1 readback on CQ1), which needs Tensix dispatch: 12x10 on the Galaxy, 11x10 on a p150. MAST3R_CQS=2 stays as a Galaxy-only opt-in. With hostcol the server uploads the views as uint8.

p150 ETH-dispatch compliance 2026-10-05 (chip 16, start = d02a5c2, final = this commit)

Requirement: on a single p150 the model must use the 12x10 grid, so dispatch must run on ETH cores (stock Tensix dispatch leaves 11x10). ETH dispatch in the patched tt-metal has 1 CQ. On this Galaxy, Tensix dispatch still gives 12x10 (13 Tensix columns), so numbers measured with it do not represent a p150. p150-equivalent here = ETH dispatch, num_command_queues=1, grid capped at 12x10. (The first attempt was cut short by host crash #5; its WIP commit 2cf38a7 was reviewed, its host test fixed and every number below measured again after the reboot.)

Audit (default of every device-open path)

path	before (d02a5c2)	after
Python API Mast3rP150.from_pretrained (mast3r_p150/device.py)	ETH, 1 CQ, 12x10 (dispatch="auto"; MAST3R_CQS unset = 1)	unchanged; MAST3R_CQS=2 now warns (Galaxy-only, 11x10 on p150)
HTTP server models/server/app.py	ttnn.open_device(**open_kwargs) = Tensix dispatch; CQs from MAST3R_CQS (code default 1)	mast3r_p150.device.open_device: ETH, 1 CQ, 12x10; /info -> device_config
tt-model.yaml serve.env (+ SERVING.md)	MAST3R_CQS: "2" -> Tensix dispatch, 2 CQ, 12x10 on the Galaxy / 11x10 on a p150	MAST3R_DISPATCH: "auto", MAST3R_CQS: "1" -> ETH, 1 CQ, 12x10
test_mast3r.py (card PCC + latency row)	Tensix; validation ran it with MAST3R_CQS=2	shared opener: ETH, 1 CQ, 12x10
eval_mast3r.py, make_demo.py --local, eval_eth3d.py	Tensix, 1 CQ (eval_eth3d: plain open)	shared opener: ETH, 1 CQ, 12x10
bench_breakdown.py (OPT_REPORT trace / e2e rows)	default --mode worker11 (Tensix, 11x10); eth12 rows used --mode eth12; served rows MAST3R_CQS=2 --mode worker	default --mode eth12 (ETH, 1 CQ, 12x10); worker modes stay as explicit A/B
tools_prof/real_pair_acc.py (real-pair gate)	Tensix unless MAST3R_ETH=1	ETH unless MAST3R_ETH=0
tools_prof/sym_check.py (pose path)	Tensix; validation ran it with MAST3R_CQS=2	shared opener: ETH, 1 CQ
test_api_device.py, test_warmup_device.py, first_call_bench.py	via from_pretrained: ETH, 1 CQ, 12x10	unchanged
profile_eager.py	default --mode eth12	unchanged

1-CQ path: the model already had one (MAST3R_CQS=1, one trace on CQ0, both heads read on CQ0 after it); only the served default used CQ1. No new device code. Fallback: without the patch (marker in tt_metal/impl/dispatch/topology.cpp) auto uses Tensix dispatch with a RuntimeWarning + log line naming the 11x10 grid; an ETH open that raises also falls back with a warning (unless MAST3R_DISPATCH=eth). MAST3R_DISPATCH=eth with MAST3R_CQS=2 is an error.

Bit identity (ETH 1 CQ vs the previous served default, Tensix 2 CQ; same chip, same session)

• bench_breakdown.py --dump (ETH) / --cmp (MAST3R_CQS=2 --mode worker): float randn pair and uint8 pair, both heads bit-identical (n_diff = 0).
• HTTP: two ETH servers vs one Tensix 2-CQ server: npz arrays (/predict and /predict_npz) and PNG bytes identical.
• test_api_device.py (ETH): model(...) == the server pipeline bit for bit, with and without pose.

Accuracy (code/tools_prof/eth_validate.sh 16; all in ETH, 1 CQ, 12x10; gate code unchanged; all gates pass)

metric	published (audit integration 2026-10-04)	now (ETH, 1 CQ)	gate
randn head1 / head2 (all), float	0.99862 / 0.99869 (0.99891)	0.99862 / 0.99869 (0.99891)	0.998
test_mast3r.py --layer end_to_end	0.9985 PASS (2 CQ)	0.9985 PASS (head1 0.9985, head2 0.9982)	0.99
real pair float pts3d h1 / h2, conf h1 / h2	0.99016 / 0.99088, 0.99269 / 0.99486	same digits	0.989 / 0.99
served-path u8 real pair pts3d h1 / h2, conf h1 / h2	0.99015 / 0.99097, 0.99246 / 0.99519 (2 CQ)	same digits	0.989 / 0.99
sym out_ii / out_ji / out_jj vs two-pass	bit-identical	bit-identical	identical
synthetic u8 head1 / head2 (not gated)	0.99593 / 0.99927	0.99593 / 0.99927

Performance before / after (chip 16; medians, min in brackets)

metric (method)	published	p150 config now (ETH, 1 CQ, 12x10)	Tensix 2 CQ, same session (Galaxy-only)
trace, float (bench_breakdown, 30 it)	23.32 ms eth12	23.42 (23.07)	23.67 (23.15)
e2e model(img1, img2), float	28.53 eth12	28.38 (27.86)	27.55 (26.89)
e2e, uint8 views (served input path)	26.27 / 26.36 (worker 2 CQ)	27.83 (27.37)	26.64 (26.36)
test_mast3r.py --layer end_to_end --runs 25	27.51 ms (2 CQ, round 9)	27.77 ms
pose: single pair / sym / two-pass e2e (sym_check)	26.40 / 36.98 / 52.48 (2 CQ)	27.22 / 38.89 / 54.90
server timing_ms.forward, npz (2 x 30 req)	25.5 / 25.57 (2 CQ)	26.55-26.98 (2 servers x 2 runs)	25.63
/predict npz total / client wall	81.5-83.3 / 130-140	80.3-82.5 / 123-136	79.4-80.0 / 127-130
png total (2 x 15 req)	124.5-139.6	131.8-143.7	123.9-124.7
/predict_npz total / client wall	72-76 / 96.8-102.0	72.7-74.0 / 97.5-104.5	71.8-72.0 / 96.8-100.4
Python API device call timing_ms['forward'] (30 calls)	26.7 (26.4)	26.99 (26.54)
model(PIL, PIL) / model(path, path) whole call	41.9 / 57.1	46.75 / 58.65
predict_pairs per pair	27.9	28.09
startup (from_pretrained, warm cache)	~9 s	7.5 s (device warm-up 5.8 s, host 1.1 s)
first call / warm same input (test_warmup_device): pair, pose, batch	0.92x, 1.08x, 0.99x	0.97x, 1.09x, 1.00x (gate 10 % + 5 ms: pass)
smoke --require-pose --npz-route	PASS	PASS x2	PASS

Reading: on the device the configurations are the same (trace 23.4 vs 23.7 ms, within noise). The p150 config loses the CQ1 overlap of the head-1 readback: the served forward is about 1.1 ms (4 %) slower than the Galaxy-only 2-CQ mode, and the uint8 e2e about 1.2 ms. Request totals move by 1-3 ms, inside the host noise. The 2-CQ numbers are not reachable on a p150 at 12x10, so the ETH column is the honest p150 number. Logs: <scratchpad>/mast3r-p150-eth-reval/ (val/*.log, http/ab_*.txt, api.json, warmup.json).

Incident during this task

Before the chip was claimed, a host-test command (pytest models/tests) also collected the two device tests and ran them for about 2 minutes (04:53-04:56 UTC) without chipenv.sh, so with every Galaxy chip visible (device 0 = UMD chip 0, not in the pool). The run reached its summary (62 passed, 2 failed) and was interrupted (SIGINT) while it closed the device. A chip-2 hang reported by another agent at 04:55 overlaps this window; a link was not proven. Host tests are now run with TT_VISIBLE_DEVICES=none and an explicit file list.

Audit integration 2026-10-04 (chip 1, start = ad39150, final = this commit)

Source: audit/mast3r-p150/AUDIT_REPORT.md (verdict REOPEN) and the audit branch audit/mast3r-p150 (9bfbb82, d2e7aee). The user decided to integrate every demonstrated item of kind A (host / serving, outputs and /predict unchanged), B (additive binary route) and C (precision change that passes every gate, default on, env switch back). Each item is re-implemented on main and measured again on chip 1 (Galaxy BH, PCIe x1, 12x10 grid). Logs and scripts: audit/mast3r-p150/integrate/ (lofi_ab.sh, http_ab.sh, http_seq.sh, r3val_new.log, ab_*.txt, smoke_*.txt, server_*.log).

Integrated

commit	item	change	measured on chip 1 (A/B against the pre-item state)
4c9303a	C: lofie, lofid	Encoder and decoder MLP fc1 / fc2 at LoFi; the same knobs upload those weights rounded to nearest-even at 4 explicit mantissa bits (LoFi reads the weights as hidden bit + 4 MSBs, so the device multiplies the rounded weights exactly; truncation alone fails the gate, 0.99364). Ported from the audit env knobs MAST3R_LOFI_MLP=both MAST3R_LOFI_RNE=1 as two MAST3R_OPT entries in all: the weight caches key on cfg, so a knob change re-uploads the weights, and MAST3R_OPT=none is unchanged (none_eth12 PCC identical to ad39150). Switch back: MAST3R_OPT=all,-lofie,-lofid (served outputs then bit-identical to ad39150, checked over HTTP)	Served path (worker, 2 CQ, uint8, 60 iterations, A/B/A/B): trace 25.45 / 25.89 -> 23.62 / 23.56 ms (-7.7 %); e2e 28.29 / 28.45 -> 26.27 / 26.36 ms (-2.06 ms, -7.2 %). eth12 (ETH dispatch, 1 CQ, float, 30 iterations): trace 25.39 / 25.38 -> 23.32 / 23.24 ms (-8.3 %); e2e 30.01 / 30.98 -> 28.53 / 27.98 ms (-7.3 %). Server forward 27.0 -> 25.5 ms. Pose (sym_check, same run): two-pass 52.48, symmetric 36.98, single pair 26.40 ms
1138587	A: stored npz	/predict writes the npz with np.savez (stored) instead of np.savez_compressed. Request field compress_npz (optional, default from env MAST3R_NPZ_COMPRESS, default 0) gives the deflated file back	30 npz requests x 4 runs per side, base = ad39150 server (2 runs, interleaved) vs new: server timing_ms.total 294.5-297.1 -> 81.5-83.3 ms (-72 %); encode 228-230 -> 18.1-18.6 ms; client wall 330.9-333.1 -> 130.0-139.7 ms (-60 %). Response 6.77 -> 9.79 MB (base64 JSON). Same server with compress_npz: true: total 294-304 ms (the old cost; isolates the item). Arrays identical after np.load (n_diff = 0 on all four arrays)
d32f521	A: parallel PNG	output_format: "png": the 4 PNGs are encoded in a 4-thread pool (MAST3R_PNG_THREADS, default 4; 1 = serial). The pool is created once under a lock (sync handlers run concurrently in FastAPI's threadpool; host test with 8 racing threads) and shut down with the app	15 png requests x 2 runs per server: new code with MAST3R_PNG_THREADS=1 total 192.1 / 192.9 ms vs default 124.5-139.6 ms (-62 ms, -34 %); vs the ad39150 server: total 195.2-200.8 -> 124.5-139.6 ms, client wall 213.9-220.1 -> 142.1-160.0 ms (-31 %). PNG bytes identical (serial vs parallel, same build)
1eb8d6a	B: POST /predict_npz	New route, same JSON request; the response body is the npz itself (application/x-npz), the other fields are compact ASCII JSON in the X-Mast3r-Meta header. /predict unchanged (same keys in the same order, host test). smoke_test.py --npz-route; host tests models/tests/test_server_host.py (9 tests)	30 requests x 4 runs: total 72-76 ms, client wall 96.8-102.0 ms (-70 % vs the ad39150 /predict npz 331-333 ms, -35 ms vs stored /predict). Body byte-identical to /predict's decoded npz_b64; arrays n_diff = 0

Accuracy (code/tools_prof/r3_validate.sh 1 at 1eb8d6a; gate code unchanged; all gates pass)

metric	ad39150 (audit base run)	now (lofie + lofid)	gate
randn head1 / head2 (all), eth12 float	0.99848 / 0.99855 (0.99875)	0.99862 / 0.99869 (0.99891)	0.998
test_e2e (2 CQ)	0.9985 PASS	0.9985 PASS (head1 0.9985, head2 0.9982)	0.99
real pair float pts3d h1 / h2	0.99035 / 0.99081	0.99016 / 0.99088	0.989
real pair float conf h1 / h2	0.99262 / 0.99511	0.99269 / 0.99486	0.99
served u8 real pair pts3d h1 / h2	0.99021 / 0.99086	0.99015 / 0.99097	0.989
served u8 real pair conf h1 / h2	0.99232 / 0.99497	0.99246 / 0.99519	0.99
sym out_ii / out_ji / out_jj vs two-pass	bit-identical	bit-identical	identical
MAST3R_OPT=none randn	0.99836 / 0.99895	0.99836 / 0.99895 (unchanged)
not gated: synthetic u8 pair head1 / head2	0.99733 / 0.99941	0.99593 / 0.99927
not gated: real pair float median |dz|/z h1 / h2 vs fp32 ref	1.17 / 1.10 %	1.24 / 1.17 %
not gated: same vs the half-pixel reference	0.37 / 0.41 %	0.40 / 0.48 %
not gated: served u8 median |dz|/z h1 / h2	1.18 / 1.09 %	1.24 / 1.16 %

Disclosure: the LoFi item is not bit-identical. Every gate passes with the same or better margin, but the ungated synthetic-u8 head1 PCC drops by 0.0014 (from the encoder group) and the median relative depth error on the real pair grows by about 0.07 percentage points per head. The audit flagged these two as "owner review"; the user decision for this stage was to adopt precision changes that pass all gates, with an env switch back (MAST3R_OPT=all,-lofie,-lofid, verified bit-identical to ad39150 on the served path).

Served smoke tests

smoke_test.py --require-pose PASS on both ad39150 servers; --require-pose --npz-route PASS on all four new-code servers (default, MAST3R_PNG_THREADS=1, old-numerics env, default again). Host tests: test_fused_host.py 32/32, test_server_host.py 9/9.

Skipped (from the audit list)

• json splice of the base64 payloads into the response body (audit d2e7aee): about -5 ms, inside the +-10 ms noise (audit: inconclusive). /predict_npz removes that cost for clients that opt in.
• par (decode + preprocess of the two views in a thread pool): measured negative / bimodal in the audit (GIL).
• fmed (np.partition median in the summary): too small to measure (audit), and the summary is outside timing_ms.
• Every device item that the audit did not demonstrate (SDPA redesign, LN fold, hrope fold, polyphase border strips, bfloat8_b K/V, HiFi2 DPT phase convs with bias correction, megakernel, concurrent DPT heads on sub-devices): not attempted in the audit, so out of scope here.
• Serving on an x8 Galaxy chip (audit #17): a placement item, not demonstrated, and not relevant on a real p150 (x16).
• Nothing integrated needs a grid other than 12x10 or Tensix-only dispatch: lofie / lofid were measured on both eth12 (ETH dispatch, 1 CQ) and the served worker 2-CQ path.

Serving files

tt-model.yaml: serve.env pins MAST3R_NPZ_COMPRESS: "0" and MAST3R_PNG_THREADS: "4", the MAST3R_OPT comment lists lofie / lofid, two new verify lines (knobs in all; /predict_npz route and compress_npz field), and the card quickstart lists compress_npz and /predict_npz. No new runtime package (stdlib json, concurrent.futures only). SERVING.md: env table, compress_npz, the stored-npz change note, and the /predict_npz contract.

Round 9 (2026-10-03, chip 9, round start = 5b918db, final = ad39150)

Verifier notes on round 8, resolved

• Span extraction script. code/tools_prof/trace_span.py (commit 0c5f9fe) is the device-span tool: per trace replay (grouped by METAL TRACE ID + METAL TRACE REPLAY SESSION ID) it prints max(FW end) - min(FW start) at 1.35 GHz, the program count, the kernel-duration sum and the summed FW gaps; --list prints the per-program timeline. Usage: MAST3R_CQS=1 python -m tracy -r -p -v -o <dir> --op-support-count 6000 tools_prof/prof_trace.py then python3 tools_prof/trace_span.py <dir>.
• Re-validation of 5b918db on chip 9 (14:26-14:30 UTC). It reproduces the verified numbers: eth12 trace 24.96 / 24.68, e2e 29.99 ms; served u8 trace 25.32, e2e 28.44 ms; all accuracy lines identical to the verified table; device span 23.531 / 23.531 / 23.545 ms (693 programs, kernel sum 22.90 ms).
• Server smoke test re-run (worker dispatch, 2 CQ, the tt-model.yaml serve env, chip 9, at e9396dd): PASS 4 / 4, ready 6.2 s after start, forward 37-39 ms per pose request, pose rot 43.2 deg, f1 / f2 446 / 440 (same as round 8).
• Host wall vs device span. Explained below (AICLK throttling). The device span stays the reliable A/B metric; every host-wall A/B in this round is reported with that caveat.
• Process. Every new kernel / kernel variant of this round ran first alone, on the smallest shape, under TT_METAL_WATCHER=2 and timeout -s INT 60/120, one device process per command. No hang, no fault.

Measured: where the 1.3-2.2 ms between trace wall and device span goes (backlog item 1)

• Fixed trace launch + completion latency is 0.02 ms (tools_prof/r9_launch_probe.py, ETH dispatch: a 1-program trace, execute + synchronize, median 0.023 ms; event_synchronize 0.019 ms; idle synchronize 0.010 ms). Tiny programs cost 6.3 us each when dispatch-bound (700-program trace of 1-core adds: 4.43 ms), which is not the case in this graph (in-trace FW gaps 0.24 ms in total).
• The chip clock drops under load. tt_aiclk (sysfs, read-only, chip 9) sampled every 50 ms during a 200-iteration bench: 1350 MHz when idle or lightly loaded, 1231-1343 MHz (mostly 1256-1287) while the forward runs at 100-170 W. The device profiler counts cycles and converts at the nominal 1.35 GHz, so the span understates wall time: tools_prof/r9_clock_probe.py (traces of 20 / 120 stock matmuls, wall slope vs cycle slope) gives an effective clock of 1.296 GHz (144 us matmuls) and 1.314 GHz (53 us matmuls). For this graph: 23.50 ms of cycles at ~1.28 GHz = 24.8 ms, which is the measured eth12 trace wall (24.76-24.96 ms). The gap is DVFS, not host or dispatch overhead; model code cannot remove it (less energy per forward would raise the clock a little).
• Served-path tail (tools_prof/r9_tail_timeline.py, worker 2 CQ, u8, host event waits as read_outputs): prep + H2D + enqueue 0.55 ms; segment ends without readbacks A 22.51 / B 25.68 / C 26.30 / D 26.92 ms; with the overlapped readbacks the last segment ends ~0.9 ms later, and the e2e is 28.0 ms. The tail after the last segment is only ~0.15 ms.
• The overlapped CQ1 readbacks stall CQ0 kernels (Galaxy x1 PCIe). tools_prof/r9_cq2_prof.py (served path under the device profiler, overlapped vs synchronize-then-read replays): segment A is unchanged (19.55 ms); segment B (head 2 main rows) goes 2.92 -> 3.44-3.54 ms and segment C 0.54 -> 1.13 ms, each because ONE program (a different one per replay: e.g. the 64^2 dadd GenericOp 24 -> 637 us, a 2-us I2S -> 565 us) stalls while the 2 MB D2H transfer runs. Reproduced with stock ops only (tools_prof/r9_d2h_stall.py, trace of 300 I2S programs + a concurrent 2 MB CQ1 read): one program stalls 645 us; all its cores start on time and finish late (their NoC transactions wait). 8 x 256 KB reads give one 80 us stall per chunk; an L1 source gives many 30-40 us stalls; total stall is the same. The PCIe link of chip 9 is x1 (32 GT/s, current_link_width 1): the device -> host writes back up into the NoC for the whole transfer (2 GB/s). A p150 (x16) drains 16x faster. Effect on the served e2e: about 1.2 ms per pair on this host; the overlap still saves ~1 ms vs reading after the last segment (2 x ~1.1 ms). No model-side fix found.

Measured: SDPA cost structure with the exact production config (backlog item 2)

Method: the stock streaming SDPA compute kernel resolved from a scratch CWD (tt-metal searches the current directory first, tt_metal/impl/kernels/kernel.cpp), with compute-only edits, separate TT_METAL_CACHE per variant, watcher first run each; tools_prof/r7_sdpa_cost.py, traced, host wall per call. Stock reader / writer and program factory unchanged. Diagnostic only (wrong outputs), nothing of this is in the model.

variant (compute kernel edit)	encoder B2 H16 q160 k512	decoder B2 H12 q224 k512
stock (copy, unchanged)	94.6 us	67.4 us
no exp	91.0	60.4
no row-sum packs	92.7	65.2
no P @ V matmul	91.1	60.8
no Q @ K^T matmul	91.2	65.1
none of the four	84.2	51.2

• Even with all matmuls, the exp and the sum packs removed, the encoder SDPA takes 84 us: the stock KV reader / chain (K and V streamed twice per core: 224 q chunks on 120 cores) is a co-bottleneck. A custom compute kernel with the stock reader can save at most ~10 us per encoder call (the planner's threshold was 15 us) and ~16 us per decoder call. Not started.
• Zone timeline (stock zones enabled, top-level zones only): the math thread is busy for the whole kernel, ~2.4 us per q row and 16-tile K chunk for Q @ K^T + max + sub/exp, ~6 us per K chunk for the P @ V drain; ~300 cycles per score tile.
• Chunk sweep, stock: encoder q160 k512 94 us is the best of q96-q512 / k256-k1024 (q352 110, q384 114, q256 160 us); decoder q224 k512 67 us is the best (q256 79, q192 118 us); exp_approx 66.5 vs 67.3 us (decoder).
• What a real gain needs: per-core K / V delivered once (multicast from the hrope writer into the SDPA cores' L1) plus a leaner compute kernel and balanced 8-9 rows per core across head boundaries. Estimate -0.7 to -1.0 ms in total, a multi-kernel redesign (hrope writer + SDPA reader + compute).

Measured: other cost splits

• hrope (tools_prof/r9_hrope_probe.py on the obs block-sharded qkv; MAST3R_HROPE_DIAG defines in the model-local kernels, default build unchanged and bit-identical): stock 29.1 us (host wall per call), no reads 25.8, no writes 23.3, no compute 25.3, none of them 11.5. No single stage dominates.
• LayerNorm (tools_prof/r9_ln_probe.py, enc shape on the obs residual): stock 20.2 us; without reads 16.1 us (tools_prof/kern/ln_reader_fast.cpp, D_NOREAD); 1 or 2 blocks per barrier 20.2-20.3, 4 or 8 blocks per barrier 27.2-27.8 us. LN is compute-bound.
• fc1 output writes (obsc probe below): fc1 with a block-sharded output is 125.7 vs 127.7 us interleaved; the output NoC writes are not a cost.

Kept steps

commit	knob	change	effect
de679b6	esplit	The final enc_norm (multi_layernorm, one tile row per core) writes view 1's and view 2's encoder outputs as two tensors (each core's writer gets its view's base address and start tile) instead of one [2, N, 1024] tensor + two DRAM slices (14.6 us each). Bit-identical (eth12 float, sym_check).	device span 23.531 -> 23.508 ms (693 -> 691 programs)
e9396dd	demb	The two decoder embeds (1024 x 1024 x 768 per view, two full-grid linears 19.6 + 20.0 us + two I2S 2.7 + 2.4 us) as ONE dual program on the top / bottom 12x5 halves (dmm machinery) writing decoder step 0's per-half block-sharded residual specs directly (29.7 us). Same in0_block_w and compute config -> bit-identical.	23.508 -> 23.498 ms (688 programs)
b117533	hrqk	hrope compute: q and k of a unit in one pass per stage (4 tiles per fp32 dest half instead of 2 x 2 tiles), same per-tile op sequence -> bit-identical. Probe enc 28.9 -> 28.2 us.	23.498 -> 23.487 ms

Bit-identity of the final commit vs the round start: bench_breakdown.py --dump/--cmp, eth12 float n_diff 0 on both heads, served u8 (worker 2 CQ) n_diff 0 on both heads (5b918db worktree dump vs 6d6e155).

Tried and rejected / measured only (round 9)

• lnx: a fewer-pass LayerNorm compute kernel (row sums as x @ J, sum of squares as the diagonal of sum x x^T, SFPU statistics in fp32, (x - mean) * rstd with a dest-reuse multiply; tools_prof/rejected_lnx/). Correct; the host emulation on the real pair's LN inputs (tools_prof/r9_ln_numerics.py, |mean|/std <= 0.57) predicted equal accuracy. On device it is slower: enc 23.7 vs 20.1 us, decoder pair 20.9 vs 17.0 us; mean |err| vs fp64 1.19e-3 vs 1.17e-3. The single-tile HiFi4 matmuls cost more than the stock reduce passes.
• obsc: fc1 writes its block-sharded output in its own packing order (column-major tiles in each shard, 7x1 subblocks) and fc2 reads it through the stock interleaved in0 sender with patched strides and a "virtual transposed" TensorAccessor (heads_rope.colmajor_ta_args, linear_descriptor(in0_colmajor=True); tools_prof/r9_obsc_probe.py). Bit-identical on the first run (watcher), but fc1 125.7 + fc2 81.5 us vs 127.7 + 80.3 us: -0.8 us per pair. Not wired in.
• LN reader with more reads per barrier: slower (above).

Final validation (code/tools_prof/r3_validate.sh 9 at 6d6e155, 15:40-15:44 UTC, 30-iteration median / min)

	none baseline (eth12, 1 CQ, float)	now: eth12, 1 CQ, float	now: worker 2 CQ, float	now: served (worker 2 CQ, uint8)
trace (host wall, synchronized)	62.21 / 62.11	24.76 / 24.45	25.08 / 24.53	25.19 / 24.56
host prep / H2D	0.88 / 1.08	1.00 / 1.03	0.76 / 1.06	0.51 / 0.62
D2H, synchronized	28.70	3.31	2.84	3.02 (overlapped in e2e)
e2e model(img1, img2)	88.55 / 84.36	29.83 / 29.62	28.49 / 28.25	27.74 / 27.35
test_mast3r.py --layer end_to_end (2 CQ, float)			27.51 ms, PCC 0.9985, PASS
real pair: single pair / sym pose / two-pass pose e2e				27.81 / 38.35 / 55.83 (min 27.30 / 37.82 / 52.96)
device span, traced (eth12 1 CQ, 3 replays)		23.491 / 23.479 / 23.490 ms (688 programs, kernel sum 22.86 ms)

Same-session A/B (abco.sh, 3 alternating pairs, mean of medians, round start 5b918db -> new): eth12 trace 24.93 -> 25.01 ms, e2e 29.51 -> 30.05 ms (new = 3feb9b3, i.e. esplit + demb); served u8 trace 25.29 -> 25.20 ms, e2e 28.06 -> 28.12 ms (new = 6d6e155). All differences are inside the host-wall noise (run-to-run medians move by 0.3-0.5 ms); the device span (-0.044 ms) is the measurable effect.

Accuracy: unchanged to every printed digit (all kept steps bit-identical): randn head1 / head2 / all 0.99848 / 0.99855 / 0.99875 (gate 0.998); test_e2e 0.9985 PASS; real pair float pts3d 0.99035 / 0.99081, conf 0.99262 / 0.99511; served u8 pts3d 0.99021 / 0.99086, conf 0.99232 / 0.99497; raw PCC vs the half-pixel reference 0.999882 / 0.999814 (float) and 0.999886 / 0.999782 (u8); sym out_ii / out_ji / out_jj bit-identical to the two-pass path. Gate code unchanged.

vs RTX 5090 (GPU_COMPARISON.md; like-for-like = the same network, bf16, batch 1, one pair)

• vs bf16 + SDPA (35.3 ms incl_h2d, 34.3 ms device-only): TT served u8 e2e 27.7-28.1 ms is 20-21 % faster; TT device span 23.49 ms (cycles at 1.35 GHz; ~24.8 ms at the throttled clock) is 28-32 % faster.
• vs torch.compile + CUDA graphs (21.1 ms served, 20.1 ms device): still out of reach; the GPU is 1.31-1.33x faster served and 1.17x (cycle span) to 1.23x (throttled wall) faster on device. Of the ~6.8 ms served gap, ~1.2 ms is the D2H stall of this host's x1 PCIe link and ~1.3-1.7 ms is the clock throttling under load (trace wall vs cycle span). The stall is specific to the x1 link; whether a p150 throttles the same way was not measured.
• Pose (sym, mode-specific, reported separately): 38.35 ms vs 2 x 35.3 = 70.6 ms for two GPU forwards.

Remaining backlog (device span 23.49 ms; ceilings measured this round)

• SDPA redesign (hrope writes K / V once per head into the SDPA cores' L1 by multicast, balanced 8-9 rows per core, leaner compute): -0.7 to -1.0 ms. A compute-only kernel is capped at ~-10 us (enc) / -16 us (dec) per call.
• LayerNorm 1.45 ms in 84 programs: compute-bound (reads ~4 of 20 us). A fewer-pass kernel built from single-tile HiFi4 matmuls is slower (lnx); a win needs a kernel that keeps the stock reduce LLKs but drops the two fp32 round trips.
• fc1 GELU epilogue (~28 us x 24 + ~26 us x 12): the pack-thread SFPU can only overlap the matmul if the final sum is in dest at the last K block, which needs fp32 dest (4-tile halves) and breaks the 7x1 / 7x11 blocking.
• Served path on this host: the D2H stall (~1.2 ms per pair) is a Galaxy x1-PCIe property; on an x8 chip (5, 13, 21,
  29. it should be ~8x smaller.
• Small items left: pemm residual straight into the obs spec (-5 us), DPT 128^2 plumbing (L1-capacity bound).

Round 8 (2026-10-03, chips 3 -> 18, round start = 96fe46e, final = this commit)

Re-validation of the round start. HEAD 96fe46e on chip 3 at 12:15 UTC reproduced round 6/7:

• eth12 trace 25.94 / 24.71 ms; PCC 0.99843 / 0.99872.
• Served u8: trace 26.53 / 25.37, e2e 29.04 / 28.78 ms.
• Real pair u8: conf h1 0.99188.

Nothing was broken, so nothing was reverted.

Measurement method (new this round: device span)

• Device span. tools_prof/prof_trace.py under tracy (MAST3R_CQS=1, 3 trace replays). From ops_perf_results*.csv, take max(FW end) - min(FW start) of one replay's programs at 1.35 GHz. Replays agree to about 0.03 ms. This is the device time without any host latency. The bench's "trace" number (execute + synchronize, host wall) moves by 0.3-0.5 ms between runs on this shared host.
• Host-side numbers. bench_breakdown.py, 30 or 60 iterations, median / min, as in earlier rounds.
• A/B. Same chip, same session, alternating runs, worktree vs working tree (tools_prof/abco.sh).

chip 18, same session	96fe46e (round start)	5fbbe2d (round 8)	change
device span, traced (3 replays)	23.884 / 23.889 / 23.913 ms (690 programs)	23.525 / 23.542 / 23.550 ms (693 programs)	-0.35 ms (-1.5 %)
device kernel sum per replay	23.25 ms	22.91 ms	-0.35 ms
eth12 1 CQ float: trace (3 pairs, mean of medians)	26.44 ms (min 25.64-25.88)	26.21 ms (min 25.40-25.55)	-0.23 ms
served (worker 2 CQ, u8): trace	26.17 ms	25.85 ms	-0.32 ms
served (worker 2 CQ, u8): e2e	28.68 ms (min 28.28-28.58)	28.53 ms (min 28.06-28.34)	-0.15 ms

Final validation: code/tools_prof/r3_validate.sh 18 at 5fbbe2d, 13:31-13:36 UTC, 30-iteration median / min.

	none baseline (eth12, 1 CQ, float)	now: eth12, 1 CQ, float	now: worker 2 CQ, float	now: served (worker 2 CQ, uint8)
trace (host wall, synchronized)	62.75 / 62.67	26.01 / 25.46	25.20 / 24.74	25.68 / 24.65
host prep / H2D	0.82 / 1.06	0.90 / 1.03	0.89 / 1.09	0.44 / 0.58
D2H, synchronized	29.21	2.81	3.75	3.07 (overlapped in e2e)
e2e model(img1, img2)	92.57 / 84.59	30.48 / 30.21	29.06 / 28.42	28.52 / 28.31
test_mast3r.py --layer end_to_end (2 CQ, float)			27.95 ms, PCC 0.9985, PASS
real pair: single pair / sym pose / two-pass pose e2e				28.57 / 39.41 / 57.00 (min 28.40 / 39.10 / 55.22)

Chip 18's host wall numbers are about 1 ms above chip 9's (round 6: eth12 trace 25.15). The device span is the chip-independent comparison.

Server smoke test: worker dispatch, 2 CQ, MAST3R_OPT=all, chip 18, at 5fbbe2d, with the tt-model.yaml serve env.

• PASS 4 times out of 4. Ready 7.8 s after start.
• forward 37-39 ms per pose request (round 6: 38-39).
• Pose rot 43.2 deg, f1 / f2 446 / 440 (round 6: 43.3 deg, 445 / 440).

Kept steps

commit	knob	change	effect
8a7b17a	resmm	Encoder residual adds folded into the proj / fc2 matmul epilogues. The model-local mm_gelu_compute.cpp gets MAST3R_RESID_CB: after the FUSE_BIAS bias add, dst += resid (dest-reuse ELWADD). The residual comes from a CB globally allocated on the residual tensor's shard (heads_rope._add_resid). The residual stream stays BLOCK_SHARDED in the obs output spec, which equals the per-core output block (row-major, 1-row subblocks, asserted). Block 0 converts the patch-embed residual once (one I2S, 5.5 us). The two LayerNorms per block read only the sum, and the final enc_norm is a plain affine LN of the sum.	Probe, traced, enc: proj + add/LN 51.5 -> 46.9 us, fc2 + add/LN 114.2 -> 110.0 us. Graph, chip 3, 3 A/B pairs: eth12 trace 26.00 -> 25.72 ms (mean of medians)
8fbb2e3	resmmd	Same for the decoder: dual proj / cproj / fc2 with per-half residual shards (obsd specs). multi_layernorm now groups reader kernels per input layout, because the two branches are sharded on different grid halves. Step 0 converts both decoder-embed residuals once (2 x I2S, 2.5-2.9 us). The last fc2 sum goes straight into dec_norm. Taps (blocks 5 / 8) copy the sum to DRAM as before.	Chip 3, 3 A/B pairs: eth12 trace 26.01 -> 25.81 ms

Accuracy (gate code unchanged; all gates pass). Neither step is bit-identical. The sum is rounded to bf16 once (fp32 dest -> one pack) instead of twice (matmul output, then the add). The residual add happens in fp32 in the matmul's dest. The matmul + bias value passes the dest -> srcA move as tf32: the partials CB of these mm32 linears is fp32, so srcA is in tf32 format.

	round start 96fe46e	round 8 (5fbbe2d)
randn head1 / head2 (all), gate 0.998	0.99843 / 0.99872 (0.99893)	0.99848 / 0.99855 (0.99875)
test_mast3r end_to_end (PCC h1 / h2)	0.9988 (0.9987 / 0.9986)	0.9985 (0.9986 / 0.9982), PASS
real pair float: pts3d h1 / h2 (gate 0.989)	0.99023 / 0.99074	0.99035 / 0.99081
real pair float: conf h1 / h2 (gate 0.99)	0.99209 / 0.99509	0.99262 / 0.99511
real pair float: raw PCC vs half-pixel fp32 ref h1 / h2	0.999863 / 0.999798	0.999882 / 0.999814
real pair float: median abs(dz)/z h1 / h2	1.26 / 1.12 %	1.17 / 1.10 %
real pair served u8: pts3d h1 / h2	0.99047 / 0.99085	0.99021 / 0.99086
real pair served u8: conf h1 / h2	0.99188 / 0.99483	0.99232 / 0.99497
real pair served u8: raw PCC vs half-pixel ref h1 / h2	0.999878 / 0.999770	0.999886 / 0.999782
synthetic u8 pair (not gated)	0.99690 / 0.99939	0.99733 / 0.99941
sym out_ii / out_ji / out_jj vs two-pass	bit-identical	bit-identical

Summary of the accuracy changes:

• Against the fp32 reference, the real-pair metrics improved or stayed equal: raw PCC vs the half-pixel reference went up on both heads, and the median depth error went down.
• The thin served-u8 conf-h1 margin widened from 0.0019 to 0.0023.
• With resmm alone (chip 3), u8 conf h1 was 0.99151. resmmd moved it back up. These 1e-4-level moves are bf16 rounding variation on one pair, as in round 5.
• The synthetic randn head2 PCC (0.99872 -> 0.99855) and test_e2e head2 (0.9986 -> 0.9982) went down slightly. Both stay well above their gates.

Tried and rejected / measured only (round 8)

• hf2h: HiFi2 for the head.0 / head.2 phase convs (HiFi3 now).
  • It is fast: eth12 trace 24.58 ms, about -1 ms.
  • It is rejected for accuracy. Median depth error vs the half-pixel reference goes from 0.37 / 0.41 % to 0.73 / 0.91 % (both head convs), or 0.51 / 0.63 % (head.0 only).
  • This matches the earlier finding recorded in the code comment (HiFi2 there was 0.26 -> 0.68 %). Not kept.
• bsln: LayerNorm of the block-sharded residual in its own layout.
  • Design: per-grid-row exchange of the token sums / sums of squares (bf16 hi + lo, so the fp32 sums are exact through identity matmuls) plus a fused SFPU normalise.
  • Correct, and more accurate than the row-per-core LN: enc mean abs error vs fp64 1.135e-3 vs 1.195e-3.
  • Slower: enc 30.7 us (multicast) / 27.9 us (unicast) vs 20.4 us. The exchange (8 KB per peer) and the many small compute passes cost more than the 64 -> 110 core spread saves.
  • The stock sharded ttnn.layer_norm on the same layout is also slower (26.0 us) and less accurate (1.54e-3).
  • Code: tools_prof/rejected_bsln/.
• gelin: fc1 bias + GELU epilogue interleaved per output subblock in the last K block.
  • Bit-identical after a fix, but no gain: enc fc1 137.4 vs 136.1 us. Per subblock the chain "pack partial (L1 acc) -> unpack + bias -> SFPU GELU" is serial on the pack thread, so the SFPU epilogue still does not overlap the matmul.
  • Adding the bias before the last K block instead (so that the GELU could overlap the last block's products) was 40 % less accurate: bf16 dest accumulation at full magnitude.
  • The first version of this kernel hung chip 3 (see Known hangs).
  • Code: tools_prof/rejected_gelin/.
• hrf: RoPE in two dest-reuse stages instead of four.
  • More accurate: mean abs error vs fp64 1.75e-3 vs 2.06e-3. This needs the srcA format set to fp32 before the dest-reuse add; otherwise the dest -> srcA move truncates to bf16.
  • But enc hrope only goes from 28.6 to 27.7 us.
  • Batching more reads per barrier is slower (ub 2 / 4 / 8: 35 / 45 / 59 us), and so is a single read in flight (28.2 us).
  • Zones show about 0.9 us of read latency per unit. hrope moves about 24 MB through the NoC per call at about 1 TB/s aggregate: it is NoC-bandwidth bound, not compute bound.
  • Code: tools_prof/rejected_hrf/.

Fresh device profile (eager, after resmm / resmmd; 717 programs, kernel sum 22.94 ms)

• Matmuls (generic_op + ttnn): enc fc1 + GELU 127.5 us x 24, enc fc2 80.3 x 24, enc qkv 54.1 x 24, dec fc1 + GELU pair 84.1 x 12, dec qkx 56.8 x 12, dec fc2 pair 52.2 x 12.
• SDPA 3.59 ms: enc 88 us x 24, dec 61 us x 24.
• DPT convs 4.25 ms + halos 0.77 ms. The biggest are the head.2 phase convs (128 -> 128 at 256^2, HiFi3, 132.5 us x 8) and the 128^2 convs (resconv 256 -> 256 113 us x 8, head.0 phase 84.7 us x 8).
• LayerNorm (now LN only, no add): enc 18.8 us x 48, dec 15 us x 36.
• hrope: enc 25 us x 24, dec 21 us x 24.

Remaining backlog (device span 23.54 ms)

• Custom SDPA compute kernel, about -0.7 to -1.2 ms (round-7 analysis: P @ V at 2x the FPU peak, normalise tail).
• hrope data movement, about 1.1 ms. It is NoC-bandwidth bound: it reads and writes all of q / k / v. Writing v (and possibly q / k) straight into the SDPA layout from the qkv matmul (a tile-remapping output writer) would remove about 1/3 of that traffic. Estimate -0.2 to -0.35 ms.
• GELU epilogue, about 0.9 ms of SFPU. It is serial on the pack thread. Only a cheaper GELU at equal accuracy, or real FPU / SFPU overlap (a different blocking with fp32 partials), would help.
• LayerNorm-only programs, 1.55 ms on 64 cores. A layout-local LN needs a much cheaper statistics exchange than bsln.
• DPT head convs. HiFi2 is not an option (accuracy). Conv blocking for the 128^2 / 256^2 phase convs was explored in rounds 3-5.
• Small items:
  • pemm writing the residual straight into the obs shard spec would save the encoder I2S (5.5 us) and let block 0 use the sharded LN.
  • The decoder-embed I2S pair (5.4 us).

vs RTX 5090 (GPU_COMPARISON.md; like-for-like = the same network, bf16, batch 1, one pair)

• Against the module in bf16 + SDPA (35.3 ms incl_h2d, 34.3 ms device-only):
  • TT served u8 e2e 28.5 ms: 19 % faster.
  • TT device span 23.5 ms vs GPU 34.3 ms device-only: 31 % faster.
• torch.compile + CUDA graphs (21.1 ms served, 20.1 ms device) is still out of reach: the GPU is 1.35x faster served and 1.17x faster on device time (TT device span 23.5 vs 20.1 ms).
• Pose (sym, mode-specific, reported separately): 39.4 ms vs 2 x 35.3 = 70.6 ms for two GPU forwards.

Known hangs (round 8)

• chip 3, 13:10-13:20 UTC: tools_prof/r8_gelin_probe.py enc 20 with the first version of the new model-local fc1 compute kernel tt/kernels/mm_gelu_inline.cpp (bias + GELU epilogue interleaved per subblock in the last K block). Cause (found by review, not by re-running): the kernel waited for the bias CB at its START, but ttnn's in1 sender/writer pushes the bias only after it has pushed every in1 K block; with 8 K blocks the in1 CB fills, the reader blocks, and the compute never consumes -> deadlock. The 2-K-block small shape fit the in1 CB and passed. Fixed by waiting for the bias in the last K block (where the stock kernel waits). The fixed kernel then ran on chip 18 (watcher, 8-K-block small shape and the enc shape, bit-identical, no hang). FAULT written via chip_pool (new chip 18). A follow-on r8_gelin_probe.py dec from the same shell loop opened chip 3 for ~3 s before it was killed. Lesson: never chain several device runs in one shell loop after a new kernel; a hang must stop the loop.

Round 7 (2026-10-03, chip 9, round start = e9c87d6): no kept speedup, chip faulted

Verifier notes on round 6 (all minor), resolved in this report:

• The independently measured round-6 A/B was -2.5 % eth12 trace (claimed -2.9 %), -2.8 % served u8 trace and -1.4 % served e2e (claimed -1.7 %). Those verified figures replace the round-6 claims. Absolute verified values at e9c87d6: eth12 trace 24.89 / 24.73 ms, e2e 30.00 / 29.35 ms; served u8 2-CQ trace 25.19 / 24.77 ms, e2e 28.13 / 27.45 ms.
• Baseline reference: the first commit 20bdff0 has no bench_breakdown.py. The baseline numbers use 81ff339, the earliest commit with the bench tooling (trace 62.2 ms). MAST3R_OPT=none at the final commit reproduces it (62.19 / 62.11 ms).
• chipenv's "expected 32 chips on the bus, found 31" warning is host-level and appeared on every run of every agent.
• The server smoke test and the tracy stage profile were not re-verified independently. They were not re-run in round 7 either, because the default path did not change.

Nothing new is enabled by default. MAST3R_OPT=all is the same knob set and code path as e9c87d6. The new code below is reachable only through explicit arguments or probes. The round-7 commit changes no default kernel source.

Measured (chip 9, traced, isolated ops)

1. SDPA cost structure (tools_prof/r7_sdpa_cost.py, stock streaming SDPA, 12x10). Varying the K length at a fixed q chunking:

• decoder B2 H12 q224 / k512: 40.8 / 67.3 / 117.6 / 214.8 us at Sk = 512 / 1024 / 2048 / 4096. That is about 300 cycles per 32x32 score tile in steady state, plus about 16 us fixed per call.
• encoder B2 H16 q160 / k512: about 317 cycles per score tile, plus about 18 us fixed.
• k256 instead of k512 costs about 4.6 us per extra K chunk.
• Other points:
  • exp_approx on/off: 93.4 vs 93.5 us.
  • head dim 32 / 64 / 128: 69 / 94 / 190 us.
  • The HiFi2 FPU floor would be about 128 cycles per score tile (2 + 2 tile products).

2. fattn: stock streaming SDPA compute with a single K chunk (exact softmax, no rescale), plus a model-local reader that keeps each head's K^T / V resident in L1, plus row-balanced work (9 rows per core instead of 10) (tt/fattn.py, tt/kernels/fattn_reader.cpp, tools_prof/fattn_probe.py):

• Correct: PCC vs fp32 0.999613 (stock 0.999613); mean |err| 0.00344 vs 0.00352.
• Slower: 120 us vs 94 us (encoder), 103 vs 87 us (decoder shape). Without the K/V loads (diagnostic, wrong output) the compute alone is 80.7 us. So the stock kernel's KV chain / multicast forwarding costs only about 14 us, and loading full K/V per core costs about 40 us.
• Not kept. A win here needs a new SDPA compute kernel, not a new schedule.

3. Zone profile of the stock streaming SDPA compute (model-local copy tt/kernels/fsdpa/ with the stock zones enabled, FSDPA_PROF=1, tools_prof/zonesum.py). Per 2-row q chunk x 32 K tiles:

• Phase 1 (Q K^T + row max + partial sub/exp): about 4 us. The Q K^T matmul runs near the FPU peak.
• Phase 2 (softmax @ V + normalise): about 12 us. The P @ V matmul alone is about 6 us, i.e. about 63 cycles per tile product, 2x the peak. With dh = 64 the output subblock is only 2 tiles wide, so every product needs about one unpack.
• Taller P @ V subblocks (h = 4) would be about 8 % faster, but the stock streaming kernel produces wrong output with them (it assumes h <= 2).
• This is the concrete target for a custom attention kernel: P @ V with V-tile reuse across 4+ q rows, and the normalise / rescale tail.

4. L1-interleaved NoC read bandwidth (tools_prof/r7_read_bw.py, 120 cores each reading 128 tiles):

• Aggregate bandwidth by reads in flight per barrier: 1 -> 707 GB/s, 2 -> 955, 4 -> 879, 8 -> 731, 16 -> 514, unthrottled -> 270 GB/s.
• The same reads split over two RISCs (reader + writer kernels, both NoCs) reach 1.66 TB/s.
• ttnn's SDPA reader already throttles to 2. Model-local readers that issue many reads before a barrier should be checked against this.

5. Fused add + LayerNorm (tools_prof/r7_ln_probe.py, encoder shape, 64 rows on 64 cores; stock-config kernel 24.7 us traced, about 20 us kernel time):

• Phase marks show the add phase is bound by reading the two operands (about 9 us). The remaining five compute passes take about 10 us.
• lnf (tt/kernels/ln_fast_compute.cpp, fewer passes: ones-matmul row sums, SFPU square with packer L1 accumulation, dest-reuse normalise): correct (mean |err| vs fp64 1.62e-3 vs 1.56e-3 stock) but slower: 28.0 vs 24.7 us. The packer L1-accumulate packs cost about 350 cycles each.
• lnsplit (operand b read by the writer on the other NoC: ln_reader_split.cpp + ln_add_writer_rb.cpp): bit-identical, slower (26.0 vs 24.7 us), because the writer serialises the b reads with the residual writes.
• Not kept, and neither is wired into the model.

Chip fault (round 7)

tools_prof/r7_ln_probe.py enc with a variant of ln_add_writer_rb.cpp that read ALL of operand b before writing any residual block deadlocked:

• The compute blocked on the full 8-tile CB 17 while the writer blocked on the full 8-tile cb_inb.
• The process ignored SIGINT and was SIGKILLed.
• A follow-on loop iteration opened the device for about 2 s before it was killed too.
• FAULT file: chipstate/chip9/FAULT.

The committed ln_add_writer_rb.cpp is the interleaved (deadlock-free, previously run) variant, with a comment warning against the hung one. Lesson for the next round: any writer that consumes a compute output must drain it inside the same loop that feeds the compute, unless the CB holds the whole row.

Round-7 backlog (unchanged priorities, with the new measurements)

• Custom SDPA compute kernel, about -0.7 to -1.2 ms.
  • Phase 2 (P @ V at 2x peak, plus normalise) is about 75 % of the per-chunk time.
  • Design: P @ V with 4-row output subblocks (V-tile reuse) and the row sum folded into the P @ V matmul (a ones column).
  • Keep ttnn's KV chain / multicast reader. It is only about 14 us of overhead.
• Add + LN, -0.3 to -0.5 ms. The add phase is read-bound (about 9 of 20 us). Two-NoC operand reads with large CBs (cb_inb and CB 17 sized to the whole row, so that no deadlock is possible) should overlap the two operand streams. A pass-fusion compute kernel is not a win with packer L1-accumulation.
• GELU epilogue overlap in fc1, about -0.5 to -0.9 ms. Not started.

Round 6 summary (2026-10-03, chip 9, HEAD 8d1c6b0)

Round start = 242bfe7. Its independent verification: eth12 trace 25.88 / 25.63, e2e 30.08 / 29.80; served u8 trace 26.18 / 25.80, e2e 28.86 / 28.50. The verifier listed no blocking problems. Its one note, the thin served-u8 conf-h1 margin (0.99188 vs the 0.99 gate), is unchanged because every round-6 step is bit-identical.

Same-session A/B, round start vs now. 3 pairs each, alternating runs. A = a git worktree at 242bfe7, B = 8d1c6b0 (tools_prof/abco.sh). Each run is bench_breakdown.py, 30 iterations, median / min.

	242bfe7 (3 runs)	8d1c6b0 (3 runs)	change (mean of medians)
eth12 1 CQ float: trace (device)	25.77 / 25.70 / 26.01 (min 25.50-25.69)	25.00 / 25.29 / 24.98 (min 24.77-24.84)	25.83 -> 25.09 ms, -2.9 %
eth12 1 CQ float: e2e	30.86 / 30.88 / 30.95	29.99 / 30.01 / 30.19	30.90 -> 30.06 ms, -2.7 %
served (worker 2 CQ, u8): trace	25.95 / 26.29 / 26.18	25.05 / 25.44 / 25.64	26.14 -> 25.38 ms, -2.9 %
served (worker 2 CQ, u8): e2e	28.76 / 28.88 / 28.86 (min 28.20-28.55)	28.22 / 28.38 / 28.46 (min 27.48-27.94)	28.83 -> 28.35 ms, -1.7 %

Final validation: one code/tools_prof/r3_validate.sh 9 session at 8d1c6b0, 10:24-10:28 UTC, 30-iteration median / min.

	none baseline (eth12, 1 CQ, float)	now: eth12, 1 CQ, float	now: worker 2 CQ, float	now: served (worker 2 CQ, uint8)
device forward (trace, synchronized)	62.22 / 62.12	25.15 / 24.68	25.10 / 24.88	25.36 / 25.09
host prep / H2D	0.91 / 1.07	0.85 / 1.02	0.89 / 1.05	0.41 / 0.58
D2H, synchronized	28.14	2.89	3.51	3.18 (overlapped in e2e)
e2e model(img1, img2)	91.66 / 84.02	29.91 / 29.48	28.91 / 27.98	28.32 / 27.46
test_mast3r.py --layer end_to_end (2 CQ, float)			28.12 ms, PCC 0.9988, PASS
real pair: single pair / sym pose / two-pass pose e2e				28.34 / 39.33 / 56.84 (min 27.91 / 38.80 / 54.96)
eager profile: programs / kernel sum	1252 / 60.97	714 / 23.29 (round start 714 / 24.19)

Server smoke test (worker dispatch, 2 CQ, MAST3R_OPT=all, chip 9, at 8d1c6b0) passed 4 times out of 4. The server was ready 6.4 s after start. forward took 38-39 ms per pose request (round 5: 39-41 ms). Pose rot was 43.3 deg and f1/f2 were 445/440, the same as round 5.

Accuracy: every round-6 step is bit-identical. Each step was compared with bench_breakdown.py --dump/--cmp (n_diff = 0 on both heads), on the eth12 float path and on the served u8 2-CQ path. The validation run reproduces the round-5 numbers to every printed digit:

• randn head1 / head2 / all: 0.99843 / 0.99872 / 0.99893 (gate 0.998).
• test_mast3r end_to_end: PCC 0.9988, PASS.
• Real pair, float: pts3d 0.99023 / 0.99074, conf 0.99209 / 0.99509, median dz/z 1.26 / 1.12 %. Raw PCC vs the half-pixel reference is 0.999863 / 0.999798.
• Real pair, served u8: pts3d 0.99047 / 0.99085, conf 0.99188 / 0.99483, median dz/z 1.17 / 1.09 %.
• Synthetic u8: 0.99690 / 0.99939.
• sym: out_ii / out_ji / out_jj are bit-identical to the two-pass path.

Matmul efficiency audit (fresh eager device profile, 12x10, HiFi2 + fp32 dest except the fc1 GELU kernels)

linear (per forward)	M x K x N	round start us (TFLOP/s)	now us (TFLOP/s)	note
enc qkv (x24)	2048 x 1024 x 3072	68.7 (188)	54.1 (238)	obs
enc proj (x24, pcat)	2048 x 1024 x 1024	29.2 (147)	24.2 (178)	obs
enc fc1 + GELU (x24, gpoly)	2048 x 1024 x 4096	127.5 (135)	127.4	about 100 matmul + about 27-30 SFPU epilogue
enc fc2 (x24)	2048 x 4096 x 1024	84.9 (202)	79.1 (217)	obs
dec qkx pair (x12, dmm)	2 x 1024 x 768 x 3840	74.6 (81)	56.9 (106)	obsd
dec fc1 + GELU pair (x12)	2 x 1024 x 768 x 3072	84.0	84.0	matmul-only 59; GELU about 26
dec fc2 pair (x12)	2 x 1024 x 3072 x 768	54.9	51.3	obsd

tools_prof/r6_mm_sweep.py sweeps the full 2-D mcast config space on 12x10:

• grid shapes, transposed mcast, per_core_M/N incl. 11-/12-column N splits, in0_block_w 2-16, subblocks, out_block_h/w splits.

It confirmed the shipped configs as the fastest for every encoder and decoder shape. The remaining loss was not in the compute configuration:

• The per-core compute runs at about 40 cycles per 32x32x32 tile product (HiFi2 peak is 32).
• With an L1-interleaved output, the in1-sender/writer finished ~15 us after the compute threads in the encoder qkv (its 56 output tiles are NoC-written to 120 L1 banks at the end).
• A BLOCK_SHARDED output equal to the per-core block lets the matmul pack straight into its own shard: qkv 75.1 -> 60.2, proj 31.0 -> 25.7, fc2 92.0 -> 87.4, decoder qkx 74.2 -> 58.4 us (in isolation, traced). That is what obs / obsd do.

The remaining grid-quantisation loss (64 tile rows on 10 grid rows -> 7 rows per core vs 6.4 ideal; fc2 N = 32 tiles on 11 of 12 columns) was not recoverable with uniform 2-D blocks. A 60 + 4 tile-row split into two programs estimates at only -5 us per fc1, because the 4-row tail job is inefficient. It was not done.

In-trace dispatch gaps are small. A traced device profile (tools_prof/prof_trace.py, 3 iterations) gives a span of 23.90 ms for the 690 programs of the main segment, with a kernel sum of 23.27 ms. So in-trace gaps total about 0.63 ms (about 0.9 us per program), not the ~1.9 ms estimated in round 5 from (bench trace - kernel sum). The rest of the bench's "trace" time is host enqueue / completion latency. Program-count reduction is therefore worth about 1 us per removed program.

Round-6 steps (A/B in one session; trace = eth12 1-CQ median)

commit	knob	change	effect
667e43e	obs	Encoder qkv / proj / fc2 write BLOCK_SHARDED outputs whose shard is the 2-D mcast per-core block (_bs_mc), so there are no output NoC writes. The last shard row may be partial: 64 tile rows = 9 x 7 + 1. Consumers: hrope reads the qkv shards (reader kind = sharded layout); the fused add + LN reads proj / fc2 through its stock TensorAccessor (compile-time args of the sharded tensor)	26.12 / 25.95 -> 25.43 / 25.45 ms (2 pairs); bit-identical
5e87349	obsd	Decoder dual (top / bottom 12x5) qkx, proj, cproj and fc2 write per-half BLOCK_SHARDED outputs. hrope takes two sharded layouts (self: a1 top / a2 bottom; cross: the other branch's qkx half). The fused add + LN builds one reader kernel per residual layout, each on the cores of its tensor's rows	25.38 / 25.72 -> 24.92 / 25.20 ms (2 pairs); served u8 trace 26.13 -> 25.31 (obs + obsd); bit-identical (eth12 float, served u8)
20ce5de	(helper)	_patch_in0_ta: a block-sharded in0 read by the stock interleaved in0 sender through a patched TensorAccessor (descriptor built on a shape-equal placeholder). Bit-identical (obsf_probe.py); used only when an in0 is sharded
8d1c6b0	(serve)	tt-model.yaml knob comment (obs, obsd; no new kernel files, the verify list is unchanged)

Round-6 tried and rejected / measured only

• obsf (encoder / decoder fc1 output block-sharded -> fc2 in0 via _patch_in0_ta): a sharded matmul output needs 1-row subblocks (a 7x1 subblock writes column-major blocks: obsf_probe.py n_diff 8.0M). The 1x1-subblock GELU fc1 runs 135 -> 151 us. Not kept.
• hrope with direct block-shard addressing (NoC table, no TensorAccessor) + decoder cq block-sharded: no gain in 2 A/B pairs (25.06 / 24.95 vs 24.94 / 25.17 ms). Not kept.
• fc1 with fp32 dest + the GELU epilogue (1x4 subblocks): 144.5 vs 135 us. fc1 out_block_w splits (overlap the epilogue of one block with the next): 151-162 us. Decoder fc1 config sweep (in0_block_w 2-24, subblocks): none faster than the shipped 84.8 us.
• LayerNorm: a 2-way width split needs 128 cores for the 64 encoder (and 2 x 32 decoder) tile rows, which is more than 120. Switching the fp32 intermediate CBs to bf16 (r6_ln_probe.py) does not change the time (24.6-24.8 us), so the stock LN compute is not CB-traffic bound. Not pursued without a new LN compute kernel.

Remaining backlog (device ~25.1 ms, eager kernel sum 23.29 ms, in-trace gaps ~0.6 ms)

• GELU epilogue, ~1.0 ms (enc fc1 ~28 us x 24, dec fc1 pairs ~26 us x 12). It is SFPU-bound on the math thread after the last K block. Only overlapping it with the FPU (a new low-level compute kernel) or a cheaper approximation at equal accuracy would cut it.
• Add + LayerNorm, ~1.85 ms (48 x 24 us enc on 64 cores, 36 x 19.5 us dec). Compute-bound in the stock multi-pass kernel. A new LN compute kernel (fewer passes; not bit-identical) is estimated at -0.5 to -0.8 ms, with the served-u8 conf-h1 margin of 0.0019 as the accuracy risk.
• SDPA, 3.6 ms (enc 88 us x 24 at ~97 TFLOP/s, dec 61 us x 24). Stock-kernel floor at q160/k512 and q224/k512; exp-approx, LoFi and other chunkings were measured earlier with no gain.
• DPT ring strips: ~0.46 ms of kernels in ~46 small programs per forward. The halos on 6-row strips take 23-25 us each. An exact border correction of the polyphase convs (1-row correction terms instead of re-running upsample + conv on strips) is estimated at -0.4 ms. It is a redesign of the phase path with rounding changes.
• Grid quantisation of the encoder matmuls (7 vs 6.4 tile rows per core): ~0.2-0.3 ms, needs non-uniform per-core blocks (a custom matmul) to recover.

vs RTX 5090 (GPU_COMPARISON.md; like-for-like = the same network, bf16, batch 1, one pair)

• Module in bf16 + SDPA, 35.3 ms incl_h2d (34.3 ms device-only):
  • TT served 28.3 ms: 20 % faster.
  • TT eth12 1-CQ 29.9-30.1 ms: 15 % faster.
  • Device only: TT 25.1 vs 34.3 ms, 27 % faster.
• torch.compile + CUDA graphs, 21.1 ms served (20.1 ms device): still out of reach. GPU is 1.34x faster served and 1.25x device-only. Closing the gap needs ~19 ms of device time, the practical floor estimated by the planner, at the gated precision.
• Pose (sym, mode-specific, reported separately): 39.3 ms vs 2 x 35.3 = 70.6 ms for two GPU forwards.

Round 5 summary (2026-10-03, chip 9, HEAD 2a5f1ed)

Round start = e845c4c (verified: eth12 trace 28.76, e2e 33.71; served u8 trace 28.98, e2e 31.91). Re-measured at the start of this session (07:44 UTC): eth12 trace 28.72 / 28.60, e2e 33.23 / 32.88; served u8 trace 28.97 / 28.78, e2e 32.01 / 31.41. Final numbers: one code/tools_prof/r3_validate.sh 9 session at 2a5f1ed, 09:31-09:35 UTC, 30-iteration median / min.

	none baseline (eth12, 1 CQ, float)	round start (07:44)	now: eth12, 1 CQ, float	now: worker 2 CQ, float	now: served (worker 2 CQ, uint8)
device forward (trace, synchronized)	62.21 / 62.08	28.72 / 28.60 (served 28.97 / 28.78)	25.85 / 25.52	26.14 / 25.67	26.23 / 25.72
host prep / H2D	0.79 / 1.07	0.80 / 1.01	0.99 / 1.03	0.87 / 1.06	0.39 / 0.56
D2H, synchronized	28.30	3.04	3.04	2.83	2.68 (overlapped in e2e)
e2e model(img1, img2)	91.73 / 91.39	33.23 / 32.88 (served 32.01 / 31.41)	30.96 / 30.78	29.34 / 28.94	28.88 / 28.16
test_mast3r.py --layer end_to_end (2 CQ, float)				28.84 ms, PCC 0.9988, PASS
real pair: single pair / sym pose / two-pass pose e2e		31.53 / 43.80 / 63.12 (verifier)			28.56 / 39.41 / 56.98
eager profile: programs / kernel sum	1252 / 60.97	754 / 26.86	~710 / ~23.9 (724 / 24.20 at 740fb2f, before the 16^2 tups)

Device -10.0 % this round (28.72 -> 25.85 ms eth12; 28.97 -> 26.23 ms served), -58 % overall vs none (62.2 ms). Served e2e 32.0 -> 28.9 ms (-9.8 %); eth12 1-CQ e2e 33.2 -> 30.7-31.0 ms (-7 to -8 %, host phases noisy). Server smoke test (worker dispatch, 2 CQ, MAST3R_OPT=all, chip 9, at 2a5f1ed): PASS x4, ready 7.8 s after start, forward 39-41 ms per pose request (round 4: 43-45 ms), pose rot 43.3 deg, f1/f2 445/440 (round 4: 443/438; the focal estimate moves by 2 px with the hf3 / gpoly / tups numerics).

Accuracy (all gates pass; gate code unchanged). Three steps change numerics (hf3, gpoly, tups); the other seven are bit-identical (bench_breakdown.py --dump/--cmp, n_diff = 0 on eth12 float and on served u8 2-CQ).

	round 4 / start (float ; u8)	now, eth12 float	now, served u8	none baseline
randn head1 / head2 (all), gate 0.998	0.99845 / 0.99868 (0.99887)	0.99843 / 0.99872 (0.99893)		0.99836 / 0.99895 (0.99909)
test_mast3r end_to_end	0.99844 PASS	0.9988 (h1 0.9987 / h2 0.9986) PASS		0.9984
real pair pts3d h1 / h2, gate 0.989	0.99065 / 0.99056 ; 0.99046 / 0.99080	0.99023 / 0.99074	0.99047 / 0.99085	0.98988 / 0.99076
real pair conf h1 / h2, gate 0.99	0.99149 / 0.99433 ; 0.99232 / 0.99478	0.99209 / 0.99509	0.99188 / 0.99483	0.99237 / 0.99439
real pair median abs(dz)/z h1 / h2	1.26 / 1.14 % ; 1.29 / 1.18 %	1.26 / 1.12 %	1.17 / 1.09 %	1.3 %
raw PCC vs the half-pixel fp32 reference h1 / h2	0.999828 / 0.999682 ; 0.999873 / 0.999748	0.999863 / 0.999798	0.999878 / 0.999770
synthetic u8 pair (not gated)	0.99589 / 0.99929		0.99690 / 0.99939 (0.99851)	0.99752 / 0.99934
sym out_ii / out_ji / out_jj vs two-pass	bit-identical		bit-identical

The thin float conf-h1 margin of rounds 3-4 (0.99149) widened to 0.99209. The u8 conf h1 moved the other way (0.99232 -> 0.99188, still 0.0019 above the gate). pts3d h1 float moved 0.99065 -> 0.99023 (gate 0.989), while the median depth errors improved (float 1.26 / 1.14 -> 1.26 / 1.12 %, u8 1.29 / 1.18 -> 1.17 / 1.09 %). For scale: switching the round-4 graph to ttnn's exact erf GELU (all,-gelut, slower) gives randn 0.99855 / 0.99844, real-pair float conf h1 0.99233 and pts3d 0.99039 / 0.99074. That is the same spread, so these 1e-4-level moves look like bf16 rounding variation on one pair, not a trend.

Round-5 steps (A/B in one session; trace = eth12 1-CQ median unless noted)

commit	knob	change	effect
e4005d2	(tests)	host mock tests: fake_ttnn gets uint8 / BufferType / to_torch(cq_id), the mock graph runs MAST3R_OPT=none (the optimised graph runs model-local generic_op kernels a shape-only fake cannot execute; it is device-gated)	test_fused_host.py 27/32 -> 32/32 pass
463160f	hf3	HiFi3 (fp32 acc) instead of HiFi4 for the two output-resolution head convs: head.0 / head.2 phase convs and their ring-strip convs. HiFi3 only drops the lo x lo mantissa partial product. Not bit-identical (n_diff 6-7 %, PCC vs previous 0.99998); real-pair metrics equal to 4-5 digits (conf h1 0.99149 -> 0.99150)	28.72 -> 28.32; served trace 28.92 -> 28.48, e2e 31.61 -> 31.04
feb7859	gpoly	fc1 GELU epilogue as a model-local erf-GELU polynomial. An identity epilogue through the same pack-side SFPU path costs nothing (enc fc1 100.5 us vs 198 us with ttnn's gelu_tanh), so the epilogue IS bound by the SFPU instruction count. The round-3 conclusion was wrong: its exp-based variant was simply not shorter. New: gelu(x) = 0.5 x + abs(x) s q(s^2), s = min(abs(x), 4.25), q a 9-coefficient weighted-LP minimax fit of (Phi(s) - 0.5)/s with s q(s^2) = 0.5 at s = 4.25 (tools_prof/gelu_fit.py). It goes in the GELU_TANH slot of a model-local copy of ttnn's matmul compute kernel (tt/kernels/mm_gelu_compute.cpp, mm_gelu_activation.hpp, mast3r_gelu_poly.h); the program descriptor is otherwise unchanged. It approximates the reference's exact erf GELU with max abs error 6.2e-5; the tanh form gelut is 4.7e-4 away. After bf16 rounding, mean abs error vs erf GELU at sigma 1 / 2 / 4: this kernel 5.655e-4 / 1.126e-3 / 2.252e-3; gelut 5.804e-4 / 1.174e-3 / 2.297e-3; correctly rounded erf 5.648e-4 / 1.122e-3 / 2.246e-3. In the graph: encoder fc1 188 -> 143 us, decoder fc1 pairs 128 -> ~96 us	28.35 -> 26.98; served trace 27.16, e2e 29.70
81a44aa	(gpoly)	two dst rows per SFPU step: each coefficient is loaded once (SFPLOADI pair) for both Horner chains and the two MAD chains interleave	26.98 -> 26.56, bit-identical
e70a7ad	qkx	decoder: n_b = LN(x_b) feeds branch b's qkv AND the other branch's cross-attention k/v (norm_y(x_b) == n_b with the affine folded), so each branch runs ONE linear with concatenated weights [qkv_b, ckv_other] (N 2304 + 1536, same in0_block_w 4 -> bit-identical); the self-attention RoPE reads at row stride 5*Ht, the cross RoPE reads k/v from the other branch's output. One dual program per step instead of two	26.6 -> 26.45 (3 A/B pairs), bit-identical
36c2b6a	(served path)	host-side event_synchronize before each CQ1 readback instead of a device-side wait_for_event on CQ1. tools_prof/read_slow_probe2.py: a pending CQ1 wait alone slows the CQ0 trace by 0.3-0.5 ms (enqueue -> done: none 27.47, wait0 27.81, wait2 28.00 ms); device-side wait + 3 reads 28.92 vs host-side 28.52 ms. MAST3R_HOSTWAIT=0 restores the old path	served e2e 29.66 -> 29.48 ms (5 of 6 A/B pairs, 60 iterations each), bit-identical
add4ceb	sdbl	ring-strip convs (head.0 / head.2 on the 6- / 8-row strips) with act + weight double buffering, height-sharded (strip_conv_probe.py: 68 -> 60 and 74 -> 69 us per strip conv)	served e2e -0.15 to -0.33 ms (3 A/B pairs), bit-identical
fd9b448	pilhs	the phase0 interleave kernel writes the head.2 phase convs' HEIGHT_SHARDED input spec directly: one NoC write per 32-row unit into its shard (tt/kernels/phase_il_rm_writer_hs.cpp). Removes the ~21 us interleaved-to-sharded copy per head	-0.03 to -0.21 ms (3 A/B pairs), bit-identical
e8f518c	tups	DPT refinenet bilinear x2 upsample (32^2 -> 64^2, 64^2 -> 128^2) TILE -> TILE in one model-local kernel (tt/kernels/ups2_*.cpp). Each output tile is a sum of <= 4 tile matmuls with 12 constant interpolation tiles (half-pixel, clamped == ttnn.upsample; all entries exact in bf16). HiFi4 + fp32 dest, so products and sums are exact and there is one bf16 rounding. It replaces untilize + I2S + halo + upsample + S2I + tilize (tups_probe.py: 64^2 x 256 136 -> 43 us, 32^2 108 -> 20 us). Not bit-identical: the stock upsample rounds in a bf16 dest, so 23 % of its outputs differ from the correctly rounded value; for this kernel it is 3.5 %. All gates equal or slightly better (above)	-0.34 to -0.57 ms (2 A/B pairs, 60 iterations); served 26.28 / e2e 28.76 ms
740fb2f	tapm	the 4 DPT tap projections (DRAM-latency-bound 1x1 linears, 19-22 us each) as ONE program on disjoint column blocks (6 / 3 / 2 / 1 grid columns for ap3 / ap2 / ap1 / ap0; ttnn's 2-D mcast descriptors with allowed_worker_cores, merged; dlin's in0_block_w)	79 -> 44 us per head (device profile), bit-identical
2a5f1ed	(tups)	the 16^2 -> 32^2 upsample too (W = 16: two image rows per input tile row, ups2h_* kernels with 4 constant tiles): replaces 7 small programs	min -0.12 ms (2 A/B pairs); gates unchanged
a4a2051	sgat	ring-strip inputs gathered straight from refinenet1's height shards by a model-local data-movement kernel (tt/kernels/strip_gather.cpp, 32-byte face-row reads, lr already transposed). The gathered strips are the carry for the strip segment. This replaces the 8 MB S2I to DRAM + untilize + 4 slices + 2 concats + permute	eth12 -0.06 to -0.10 ms (2 pairs), served trace -0.15 to -0.40 ms (3 pairs), bit-identical
e70a7ad..2a5f1ed	(serve)	tt-model.yaml: knob list + verify list for the new model-local kernels (mm_gelu_compute.cpp, mm_gelu_activation.hpp, mast3r_gelu_poly.h, phase_il_rm_writer_hs, ups2_*, ups2h_*, strip_gather)	verify command passes on the host

Round-5 tried and rejected / measured only

• LayerNorm math fidelity HiFi3 / HiFi2 (the 84 add+LN programs, ~1.95 ms on 64 cores): no time change (28.35 / 28.31 / 28.21 ms within noise) and the outputs change; the LN is not bound by its multiplies. Not kept.
• hrope2 (q and k of a unit batched per RoPE stage, bit-identical; tools_prof/rejected_hrope2/): within noise in 3 A/B pairs.
• Phase-conv configs (tools_prof/phase_cfg_probe.py, HiFi3): act_block_h 32-128 and act / weight double buffering are bit-identical but within ~4 us per conv. Auto act_block_h for the head.0 phase convs: 0.0-0.1 ms in the graph (noise). Merging the head.0 phase convs into 2 x 256 or 1 x 512 output channels (p0_merge_probe.py): 16 % faster at 32^2, but at 128^2 every merged variant throws the L1 / CB clash.
• cdbl for the layer_rn convs (in_c != out_c, MAST3R_FDBL A/B): bit-identical, no gain in 3 pairs.
• gpoly variants (rejected_gelu/gelu_fast_probe.py MAST3R_GELU_EXP): 7 / 8 / 9 coefficients give enc fc1 145 / 148 / 155 us (one row per step). 9 is kept for accuracy; 8 has max error 1.1e-4 and would be about -0.07 ms. Hoisting 4 coefficients into LREGs makes the SFPU compiler run out of registers. The full-graph 8-coefficient variant had randn 0.99835 / 0.99846; the 9-coefficient one 0.99838 / 0.99868.
• fc1 program config sweep with the gpoly epilogue (tools_prof/fc1_gpoly_sweep.py): the current configs stay best.
• Served tail: with the readbacks overlapped, each 2 MB CQ1 read still costs the CQ0 trace ~0.5-0.7 ms (read_slow_probe.py), and a read after the trace costs ~1.36 ms. Only the device-side event wait was avoidable (above).

Remaining backlog (device 25.9 ms, kernel sum ~23.9 ms, ~710 programs)

• Add + LayerNorm, ~1.95 ms (48 x 23 us encoder, 36 x 20 us decoder) on 64 cores. There are only 64 tile rows, every row must stay on one core for a bit-identical reduction, and 16-row face splits give 128 units for 120 cores. A width split needs a cross-core (semaphore) reduction and changes the summation order. Estimated -0.4 to -0.6 ms, with accuracy and hang risk.
• The 128^2 add3s reads the upsampled x from DRAM; writing x into the add3s shard spec would save about 0.05 ms but keeps 8 MB in L1 across the resconv convs (CB risk).
• head.2 tail: the 4 per-phase 1x1 linears write 87 % padding (4 valid of 32 columns each), and tailf re-reads 16 MB. A fused phase-lin + tail kernel would save about 0.1 ms but needs the 4 phase-conv outputs live together in L1.
• Dispatch gaps: ~1.9 ms (trace 25.9 vs kernel sum ~23.9 ms).
• Served tail: ~2.5 ms (prep 0.6, H2D 0.6, overlapped readback slowdown ~1.0, last read). It is bounded by the x1 link and the CQ0 slowdown during CQ1 reads.

vs RTX 5090 (GPU_COMPARISON.md; like-for-like = the same network, bf16, batch 1, one pair)

• Module in bf16 + SDPA, 35.3 ms incl_h2d (34.3 ms device-only):
  • TT served 28.9 ms: 18 % faster.
  • TT eth12 1-CQ 31.0 ms: 12 % faster.
  • Device only: TT 25.9 vs GPU 34.3 ms, 25 % faster.
• bf16 module 42.3 ms: TT served is 32 % faster.
• torch.compile + CUDA graphs, 21.1 ms (20.1 ms device): still out of reach. GPU is 1.37x faster (served) / 1.29x (device only).
• Pose (sym), reported separately as mode-specific: 39.4 ms vs 2 x 35.3 = 70.6 ms for two GPU forwards.
• gpoly evaluates the same erf GELU the GPU reference runs, more accurately than the earlier tanh form. tups computes the same half-pixel bilinear upsample the port always used, with exact products. Neither is an algorithmic shortcut, so both stay in the like-for-like comparison.

Round 4 summary (2026-10-03, chip 9, HEAD 8af42e2)

Round start = c9679bd (verified: eth12 trace 30.45 / 30.23, e2e 35.71 / 35.49; served trace 30.55, e2e 33.16 / 32.81). Final numbers: one code/tools_prof/r3_validate.sh 9 session, 07:29-07:33 UTC, 30-iteration median / min. The script now takes the chip and the checkout as arguments: r3_validate.sh [CHIP] [CHECKOUT_DIR].

	none baseline (eth12, 1 CQ, float)	round start c9679bd (verified)	now: eth12, 1 CQ, float	now: worker 2 CQ, float	now: served (worker 2 CQ, uint8)
device forward (trace, synchronized)	62.20 / 62.12	30.45 / 30.23 (served 30.55 / 30.34)	28.81 / 28.62	28.91 / 28.72	28.90 / 28.75
host prep / H2D	0.61 / 1.04	0.93 / 1.02	0.87 / 1.04	0.90 / 1.07	0.57 / 0.57
D2H, synchronized	20.89	3.59	3.20	3.24	3.12 (overlapped in e2e)
e2e model(img1, img2)	92.72 / 84.50	35.71 / 35.49 (served 33.16 / 32.81)	33.98 / 33.09	32.05 / 31.83	31.99 / 31.48
test_mast3r.py --layer end_to_end (2 CQ, float)		33.67		31.07 ms, PCC 0.99844 PASS
real pair: single pair / sym pose / two-pass pose e2e		32.89 / 46.22 / 65.99			31.49 / 43.46 / 62.66
eager profile: programs / kernel sum	1252 / 60.97	905 / 28.48	803 / 27.20 (at 20b64b5; pcat/p1hs remove ~50 more)

Device -5.4 % this round (30.45 -> 28.81 ms eth12; 30.55 -> 28.90 ms served), -54 % overall vs none. The served e2e moves 33.2 -> 31.4-32.0 ms between same-day runs (07:26 A/B run 31.44 / 31.07; validation run 31.99 / 31.48). That is -3.5 % to -5 %. The eth12 1-CQ e2e moves 35.7 -> 34.0 ms (-4.8 %). Server smoke test (worker dispatch, 2 CQ, MAST3R_OPT=all, chip 9, at 8af42e2): PASS x4. Pose rot is 43.3 deg and f1/f2 are 443/438, the same as round 3. forward is 43-45 ms per pose request (round 3: 46-48 ms).

Accuracy: every round-4 step is bit-identical. Each step was A/B'd with bench_breakdown.py --dump/--cmp on the eth12 float path and on the served uint8 2-CQ path; the n_diff count was 0 for both heads. The gated numbers equal round 3's to every printed digit:

• randn head1 / head2 / all: 0.99845 / 0.99868 / 0.99887 (gate 0.998).
• test_mast3r end_to_end: PCC 0.99844, PASS.
• Real pair, float: pts3d 0.99065 / 0.99056, conf 0.99149 / 0.99433, median dz/z 1.26 / 1.14 %, raw PCC vs the half-pixel reference 0.999828 / 0.999682.
• Real pair, served u8: pts3d 0.99046 / 0.99080, conf 0.99232 / 0.99478, median dz/z 1.29 / 1.18 %.
• Synthetic u8 (not gated): 0.99589 / 0.99929.
• sym: out_ii, out_ji and out_jj bit-identical to the two-pass path.

The thin conf h1 float margin (0.99149 vs the 0.99 gate) is unchanged. This round did not touch numerics.

Round-4 steps (A/B in one session; trace = eth12 1-CQ median unless noted)

commit	knob	change	effect
12205a0	(tooling)	r3_validate.sh [CHIP] [CHECKOUT]	fixes the verifier's "hardcoded chip 9 / main checkout" note
27f59fc	rchain	DPT resConfUnit conv1 -> conv2 on ttnn's L1 conv path. conv1's sharded output feeds conv2 directly, which removes an S2I to DRAM + I2S per resconv and the DRAM-path NHWC reshape copy at 16^2. At >= 64^2 the relu input is height-sharded into the conv's own spec and freed after halo (that is the only way 128^2 fits the L1 CB budget). Below 64^2 it lives in L1 interleaved. Per-resconv probe (rchain_probe.py): 128^2 376 -> 322 us, 64^2 148 -> 128, 16^2 76 -> 47	30.46 -> 30.05 ms
6850a3b	dlin	refinenet out_conv (1x1) with its input in L1, and output in L1 below 128^2. Uses an explicit full-grid config with the auto config's in0_block_w and compute config, so every output tile sees the same K sequence (dlin_probe.py: 16384x256x256 96 -> 30 us)	30.06 -> 29.86
bd8aaef	dfront	DPT tap projections write L1, so the ConvTranspose / ap3_down / layer_rn convs use the L1 conv path (removes 16^2 reshape copies). The ap1 ConvTranspose output goes to DRAM: l2_rn on its sharded output picks another config and is not bit-identical	29.96 -> 29.82
ab3928d	dshard	>= 64^2: the refinenet adds read conv2's height shards in place and write relu(s) straight into the next conv1's shard spec. Model-local add3s / add2 kernels, same FPU adds. The leading relu writes the shard spec directly (add3s_probe.py at 128^2: S2I + add3 + I2S 156 -> 72 us, S2I + add 77 -> 29 us)	29.76 -> 29.34; served e2e 31.98
cb12ce3	ups1	upsample output staged in L1 before the tilize	29.36 -> 29.27 (2 A/B pairs)
108afa4	tailf	head tail after the 4 per-phase 1x1s as ONE model-local program: phase sum + bias + WH transpose (compute) and the 16-bit NCHW interleave (writer). Replaces 3 adds + transpose + bias add + slice + untilize + tail_il (tailf_probe.py: 113 -> 55 us per head)	29.27 -> 29.18 (3 A/B pairs)
20b64b5	pemm	patch-embed linear with an explicit config (auto in0_block_w) writing the L1 residual directly (drops the 4 MB copy)	29.17 -> 29.12
ba4971d	pcat	attention output projections (encoder proj, decoder self / cross proj) read the SDPA output directly. Built from ttnn's 2-D mcast matmul descriptor with the in0 sender reader swapped for a model-local copy that remaps tile ids (tt/kernels/mm_in0_heads_reader.cpp), so 48 concat-heads programs go away (pcat_probe.py: enc 39.9 -> 33.3 us, dec pair 31.1 -> 26.7 us)	29.16 -> 28.90
194c1a0	p1hs	refinenet1's out_conv writes head.0's height-shard spec directly; the DRAM copy for the strips is one S2I. dlin uses the auto 1-D layout for the N=96 tap projection	28.91 -> 28.73
8af42e2	serve	tt-model.yaml: knob list + verify list for the new kernels (add3s/add2/tailf/mm_in0_heads)

Round-4 tried and rejected

• thalf (served path): head 2's main rows in two trace segments (rows 2i | 2i+1, tailf in half mode, exact), so the first 1 MB readback overlaps the second half. In 3 A/B pairs the e2e did not move (31.60-31.98 vs 31.68-32.03 ms). The e2e_tail_probe.py / e2e_timeline2.py timelines show why. With no reads, the device is done at 30.3 ms (including prep + H2D). With concurrent CQ1 readbacks the last trace ends about 1 ms later, so overlapped D2H on the x1 link slows the device instead of hiding behind it.
• strip1: both heads' ring strips in one last segment with one readback. e2e 31.66 -> 32.17 ms (worse).
• r1 out_conv output in L1 (instead of DRAM): CB clash in the 2-CQ served path (p1 is carried across trace segments).
• Device im2col for the uint8 upload: the host copy is only ~0.2 ms either way (measured), so it does not pay.
• Decoder 4-way cq+ckv (or qkv+ckv) merge: not done. These pairs already run at 120-160 TFLOP/s on their halves, so a merge saves only one 2 us dispatch per step (25 us per forward).

Remaining backlog (device 28.8 ms, kernel sum ~27 ms, ~750 programs)

• Encoder / decoder fc1 GELU epilogue: about 100 us x 24 + 70 us x 12, about 3.1 ms. This is the largest item. It is SFPU-bound on the pack thread. Only an accuracy-gated activation change could cut it (round-3 rejection notes).
• LayerNorm programs (add + LN) run on 64 cores, because there are only 64 tile rows. That is 48 x 23 us (enc) + 36 x 20 us (dec), about 1.8 ms. A width-split LN with a cross-core reduction could reach ~120 cores, but it is not bit-identical. Estimated -0.5 ms.
• DPT ring strips: about 0.25 ms per head across ~25 programs. Halos at 23-25 us come from 6-row, 256-wide strips on 96 cores. Possible fixes: a transposed or width-sharded layout, or one batched program. Estimated -0.2 ms.
• DPT 128^2 plumbing: tilize 35 us, relu 23 us, add3s 70 us (DRAM-bound) per head. Keeping the upsampled x in L1 would need the refinenet split moved (do u1's convs before the upsample). Estimated -0.1 ms.
• Dispatch gaps: about 2.0 ms (2.5 us x ~750 programs).
• Host mock tests (code/models/tests/test_fused_host.py): 5 tests already fail at c9679bd and still fail. fake_ttnn lacks uint8 and the newer generic_op paths. These tests are not device gates, but they are stale.

vs RTX 5090 (GPU_COMPARISON.md)

• bf16 + SDPA module 35.3 ms (34.3 device-only): TT served 31.4-32.0 ms (9-11 % faster); eth12 1-CQ 34.0 ms (4 % faster, previously a tie); device-only TT 28.8 ms vs 34.3 ms (16 % faster).
• bf16 module 42.3 ms: TT served 24-26 % faster.
• torch.compile + CUDA graphs 21.1 ms (20.1 device): still out of reach. GPU is 1.5x faster.
• Pose (sym): 43.5 ms vs 2 x 35.3 = 70.6 ms for two GPU forwards (mode-specific; reported separately).

Round 3 summary (2026-10-03, chip 9, HEAD aae6921)

Round start = b1f4930, the verified round-2 result. All numbers below are 30-iteration median / min from one session (code/tools_prof/r3_validate.sh, 2026-10-03 06:21-06:28 UTC), except the round-start column, which was measured at the start of this session (04:28 UTC served path; the eth12 round-start trace/e2e is the all,-sdpa2 A/B run of 04:4x, i.e. the b1f4930 graph).

	baseline MAST3R_OPT=none (eth12, 1 CQ, float)	round start b1f4930	now: eth12, 1 CQ, float	now: worker 12x10, 2 CQ, float	now: served path (worker 12x10, 2 CQ, uint8)
device forward (trace, synchronized)	62.18 / 62.08	39.73 / 39.57 (eth12); 39.93 / 39.69 (served)	30.44 / 30.21	30.49 / 30.35	30.59 / 30.41
host prep	0.81		0.88	0.80	0.62
H2D	1.05		1.04	1.05	0.59
D2H, synchronized, incl. host assembly	20.63		3.59	2.96	3.76 (overlapped in e2e)
e2e model(img1, img2)	92.67 / 83.87	45.16 / 44.56 (eth12); 42.43 / 42.09 (served)	35.60 / 35.09 (previous run 35.53 / 34.78)	33.77 / 33.41	33.42 / 33.13 (previous run 33.39 / 32.91)
real pair (kitchen, sym_check.py) single-pair e2e					33.01 / 32.57
return_pose (sym B-mode), real pair e2e		60.41 / 60.07 (round 2)			46.01 / 45.62 (two-pass: 65.77 / 65.07)
test_mast3r.py --layer end_to_end (2 CQ, float)		42.69 (round-2 verification)		33.58 ms
eager profile: programs / kernel sum	1252 / 60.97	1232 / 37.64	925 / 28.68 ms (at 3627397; dadd/addln2 remove ~22 more)

Device 39.7 -> 30.4 ms (-23 %) this round, 62.2 -> 30.4 ms (-51 %) overall. e2e (served path) 42.4 -> 33.4 ms (-21 %); eth12 1-CQ like-for-like 45.2 -> 35.6 ms (-21 %). Host-side phases (prep, D2H, e2e) move by up to +-0.5 ms between runs on the shared host.

Accuracy (all gates pass; gate code in test_mast3r.py unchanged since the baseline commit)

	round 2 (b1f4930) float / u8	now, eth12 float	now, served u8	none baseline
randn pair head1 / head2 (all) — gate 0.998	0.99846 / 0.99870 (0.99890)	0.99845 / 0.99868 (0.99887)	n/a	0.99836 / 0.99895 (0.99909)
test_mast3r end_to_end PCC (h1 / h2)	0.9984 (0.9984 / 0.9980)	0.9984 (0.9985 / 0.9982), PASS		0.9984
real pair pts3d PCC h1 / h2 — gate 0.989	0.99046 / 0.99081 ; 0.99043 / 0.99091	0.99065 / 0.99056	0.99046 / 0.99080	0.98988 / 0.99076
real pair conf PCC h1 / h2 — gate 0.99	0.99182 / 0.99459 ; 0.99220 / 0.99481	0.99149 / 0.99433	0.99232 / 0.99478	0.99237 / 0.99439
real pair median abs(dz)/z h1 / h2	1.13 / 1.18 % ; 1.20 / 1.21 %	1.26 / 1.14 %	1.29 / 1.18 %	1.3 %
real pair raw PCC vs half-pixel fp32 ref h1 / h2	0.999837 / 0.999849 ; 0.999879 / 0.999884	0.999828 / 0.999682	0.999873 / 0.999748
synthetic smooth uint8 pair head1 / head2 (all) (not a gate)	; 0.99711 / 0.99929 (0.99855)		0.99589 / 0.99929 (0.99740)	0.99752 / 0.99934 (0.99900)
sym B-mode out_ii / out_ji / out_jj vs two-pass	bit-identical		bit-identical

Honest notes on accuracy:

• Most round-3 steps are bit-identical (hrope, hcat, mln, addln, resl1d, cdbl, pil, til, dmm, the common-args change; each A/B'd with bench_breakdown.py --dump/--cmp). The numerics changed only through sdpa2 (larger SDPA chunks: fewer online-softmax rescales), mc2d/mc2dr (2-D mcast linears instead of minimal_matmul / the fused dit residual op) and mm32 (fp32 dest accumulation in those linears).
• mc2d/mc2dr alone (bf16 dest, larger in0_block_w) worsened the served-path real pair: conf PCC h1 0.99071, median depth error 1.43 %. Probing showed that a larger in0_block_w accumulates more K tiles in the 16-bit dest (tools_prof/resid_acc_probe.py: mean |err| vs fp64 9.1e-3 / 13.1e-3 at in0_block_w 8 / 16 vs 7.8e-3 for the old dit op). mm32 (fp32 dest, out subblocks <= 4) gives 6.0e-3 — more accurate than the round-2 matmuls — at the same speed, and it restored the real-pair gates (conf h1 0.99232). It is part of the default set.
• Remaining drift vs round 2: the real-pair median relative depth error is 1.26 / 1.14 % (float) and 1.29 / 1.18 % (u8), vs 1.13 / 1.18 % and 1.20 / 1.21 % in round 2, and vs 1.3 % for the none baseline. Head-2's raw PCC vs the half-pixel reference fell from 0.99985-0.99988 to 0.99968-0.99975. These metrics moved by a similar amount (±0.1 % median, ±0.0002 PCC) under every individual numeric knob in A/B runs (e.g. SDPA k-chunk 256 vs 512, or removing sdpa2), so they look like bf16-rounding variation on one image pair rather than a systematic loss, but they are reported as measured.
• The synthetic smooth uint8 pair (not a gate) moved 0.99711 -> 0.99589 on head 1. Its value jumps non-monotonically with the knob set (0.99604 without the mc2d family, 0.99680 without mm32, 0.99675 without sdpa2, 0.99711 with only the round-2 knobs); none gives 0.99752.

vs RTX 5090 (GPU_COMPARISON.md; "incl_h2d" = upload of both views + forward + readback of both maps)

GPU variant	GPU incl_h2d (excl_h2d) ms	TT served path (2 CQ, uint8)	TT eth12 1 CQ float (like-for-like p150 config)
fp32 strict	103.4	33.4 (TT 3.1x faster)	35.5 (2.9x)
bf16 autocast	63.8	33.4 (1.9x)	35.5 (1.8x)
module converted to bf16	42.3 (41.3)	33.4 (TT 21 % faster)	35.6 (TT 16 % faster)
module bf16 + SDPA	35.3 (34.3)	33.4 (TT 5 % faster; min 33.1)	35.5-35.6 / min 34.8-35.1 (median 0.6-0.8 % slower, min 0.6-1.5 % faster: a tie)
bf16 module + torch.compile + CUDA graphs	21.1 (20.1)	33.4 (GPU 1.6x faster)	35.5
device only (TT trace vs GPU excl_h2d)	34.3 (bf16 + SDPA)	30.6 (TT 11 % faster)	30.4 (TT 11 % faster)

Caveats: the GPU rows upload fp32 views (the TT served path uploads uint8, 1.5 MB); with uint8 input the GPU would save part of its ~1 ms of transfers, so the bf16 + SDPA comparison is a near tie in transfer-neutral terms (device-only: TT 30.4-30.6 vs GPU 34.3 ms). The compiled-graph GPU row (21.1 ms) stays out of reach at the gated precision. Pose requests: 46.0 ms with the sym B-mode vs 2 x 35.3 = 70.6 ms for two GPU forwards (mode-specific saving; reported separately). Served-like totals (PNG decode + npz encode, ~180 ms of host work) are host-bound on both accelerators.

Server smoke test (worker dispatch, 2 CQ, MAST3R_OPT=all, chip 9, at 3627397 and again at aae6921): PASS x4 each, ready 6.5 s after start (warm-up captures both traces), forward=46-48 ms per pose request (the sym graph), pose rot 43.3 deg, f1/f2 443/438 in both runs.

Round-3 steps (each kept step committed; A/B in one session; bit-identical unless noted)

commit	knob	change	effect (trace, eth12 1 CQ)
5f2e4cc	sdpa2	per-shape SDPA chunks balanced for 120 cores (flat BHq-chunk scheduling): encoder q160/k512 (7 q-chunks per head -> 224 chunks <= 2 per core), decoder q224/k512 (5 per head -> 120 chunks, 1 per core); the kernel pads the partial last chunk. Probe (sdpa_probe3.py): 133 -> 94 us, 93 -> 67 us	39.73 -> 38.69; numerics change (randn 0.99851 / 0.99867)
8668258	hrope	model-local generic_op kernel (tt/kernels/heads_rope_*.cpp): split heads + RoPE(q, k) in one program on 120 cores, reading the per-branch qkv / q / kv matmul outputs directly (also removes the decoder's batch concats); the compute kernel runs rotary_embedding_llama's exact op sequence. Probe: 72.4 -> 26.5 us (enc), 74.3 -> 24.2 / 75.3 -> 24.0 us (dec self / cross). Also: the encoder's L1 ones vector became a per-pass copy, so no L1 buffer sits under the DPT conv CBs	38.33 -> 35.81
2f6e591	hcat	model-local concat-heads kernel (pure data movement, 120 cores); the decoder writes the two branch outputs directly (no batch slices). 13.5 -> 9.6 us enc, 19.4 -> 8.3 us dec	35.81 -> 35.50
3852783	mln	the decoder's branch LayerNorm pairs as ONE program: ttnn's own LN kernels with the exact compile-time config of its factory, via generic_op, each core's runtime args pointing at its branch's tensor (64 cores instead of 2 x 32). 30.9 -> 21.3 us per pair	35.50 -> 35.10
caad9ac	mc2d, mc2dr	plain and residual linears as full-grid 2-D mcast ttnn.linear with per-shape in0_block_w (8 / 16 for K = 768 / 3072 / 4096) instead of minimal_matmul / the fused dit op (residual as a separate add): dec 768x768 25 -> 12.4 us, fc2 56 -> 33 us, enc fc2 115 -> 85 us	35.10 -> 34.33 (mc2d) -> 32.97; numerics change, see mm32
795536d	resl1d	decoder residual streams in L1 (block-5/8 taps copied to DRAM, dec_norm out in DRAM)	32.97 -> 32.52
eae74ab	addln	every residual add that feeds a LayerNorm runs inside the LN program: ttnn's FUSE_PRE_ADD reader + a model-local copy of layernorm.cpp that also packs the bf16 sum (+ a model-local writer); cb_x kept bf16 so the LN sees the same bf16 sum ttnn.add produces (bit-identical). Dec pair 41.9 -> 24.6 us, enc 54.1 -> 31.2 us	32.52 -> 32.06
d1fb6ac	cdbl	DPT 3x3 convs <= 64^2: activation + weight double buffering (64^2 height-sharded, 32^2 / 16^2 block-sharded): 96 -> 76, 51 -> 40, 45 -> 39 us	32.06 -> 31.83
644b5c5	pil	phase0 interleave by a model-local data-movement kernel on ROW_MAJOR pixel rows (the head.0 phase convs emit ROW_MAJOR) instead of concat + transpose + 0/1 permutation matmul + transpose: 30 us vs about 200 us per head	31.83 -> 31.53
ed25302	til	head-tail phase interleave by a model-local kernel (16-bit element interleave straight into the NCHW rows) instead of RM reshape (65 us) + permute + reshape + tilize + 0/1 matmul + untilize: 35 us vs about 130 us per head	31.53 -> 31.29
784b1d2	mm32	mc2d / mc2dr linears with fp32 dest accumulation (out subblocks <= 4)	same speed; restores accuracy (above)
2c106b6	dmm	the decoder's same-shape branch linears (qkv, proj, cq, ckv, cproj, fc1+GELU, fc2) as ONE program each: ttnn's own 2-D mcast matmul descriptors (MatmulMultiCoreReuseMcast2DProgramFactory.create_descriptor) placed on the top / bottom 12x5 halves of the grid with allowed_worker_cores, merged with ttnn.merge_program_descriptors, run through generic_op. Pairs: qkv 60 -> 51, 768x768 30 -> 25, ckv 44 -> 37, fc1 154 -> 132, fc2 72 -> 63 us	31.44 -> 30.64
3627397	(hrope)	shared addresses as common runtime args	no change
9082ce4	dadd	DPT refinenet with skip: x + (skip + c2) and the next unit's leading relu as one model-local program (FPU adds in ttnn.add's order + SFPU relu): 128^2 165 -> 104, 64^2 56 -> 36, 32^2 21 -> 11 us	30.65 -> 30.49
aae6921	addln2	encoder / decoder last add + enc_norm / dec_norm (affine) as the fused add + LN program with gamma/beta (stock FUSE_GAMMA/BETA path); decoder block-5/8 taps copied from the next step's fused sums	30.49 -> 30.43

Round-3 tried and rejected

• Faster GELU epilogue (x / (1 + exp(-2u)), exp_21f or fp32-accurate exp + Newton reciprocal) in a model-local copy of the fc1 matmul compute kernel, built from ttnn's own matmul descriptor (tools_prof/rejected_gelu/): 191-207 us vs 194-210 us stock; the GELU epilogue (about 96 of 198 us in the encoder fc1) is not bound by the SFPU instruction count. The exp_21f variant was also less accurate (up to 1.8 half-ulp on x in [-5, 1)).
• Separate GELU op after a plain fc1: 102 + 110 us vs 198 us fused.
• fc1 out-subblock / in0_block_w / grid sweep (fc1_sub_probe.py): best = current.
• Merging the 4 head-tail phase convs into one 128 -> 512 conv: fits L1 only at 64^2 (129 -> 95 us there); at 256^2 the height-sharded variants throw a CB/L1 clash and the block-sharded ones are 5-8x slower.
• Double buffering for the phase / phase0 convs and for the 128^2 refinenet convs: no gain.
• TILE-layout phase0 interleave reading 32-byte face rows over the NoC: 144 us (vs 30 us for the ROW_MAJOR kernel).
• SDPA q64 / q32 / q96 / q192 / q256 / q512 and k1024 variants: slower (sdpa_probe3.py).
• minimal_matmul grid / block sweep for the 768-wide decoder linears (mm_grid_sweep.py): at most 2-3 us; superseded by mc2d.
• Half-grid decoder linears split by columns (6x10 halves): only 2-4 us per pair; the row split (12x5) is what dmm uses.

Megakernel / parameter taxonomy status (round 3)

• No parameter classification changed: A (launch-fixed) = resolution 512, batch 1, weights, MAST3R_CQS, uint8 input, the knob set; B = return_pose (precompiled sym graph, captured at warm-up); C = pixel contents.
• Fusion: 1232 -> about 903 programs per forward. Model-local generic_op kernels: heads_rope, heads_concat, ln_add (copy of layernorm.cpp + writer), phase_il_rm, tail_il, add3; plus ttnn's own LN and matmul kernels re-launched through generic_op with per-core tensor bindings / merged disjoint-core programs (mln, addln, dmm). Still one trace per forward segment, one H2D, readbacks overlapped on CQ1.
• The per-program dispatch gap is now about 2.1 us (trace 30.65 vs kernel sum 28.68 ms over 925 programs): fewer programs remain the main lever for the gaps.

Remaining backlog (round-3 view; device 30.7 ms, kernel sum 28.7 ms)

• Encoder fc1 + GELU_TANH: 188 us x 24 = 4.5 ms; the GELU epilogue is ~96 us of it and is not reducible with fewer SFPU instructions (above). Only a different, accuracy-gated activation form could cut it.
• DPT: the remaining resconv add (sum + c2' before out_conv) could fold into the out_conv input path; conv in/out reshards (I2S / S2I / halo, 1.8 ms total) are bound by the stock conv L1 CB sizes.
• Head-tail post-conv chain (4 x 128->4 linears, 3 adds, transpose, bias add, slice, untilize: about 125 us per head): a fused kernel could save about 0.2 ms.
• e2e tail (served path): e2e - trace = 2.6 ms, of which about 1 ms is host prep + H2D and the rest is mostly the 2 MB head-2 main-row readback that the remaining strip work (about 0.9 ms) cannot fully hide on the x1 link.
• Compiled-GPU parity (21.1 ms) is not reachable at the gated precision.

Round 2 summary (2026-10-03, chip 9, HEAD f6cbac0)

Round start = aff19e9, the verified round-1 result. Same-session numbers, all 30-iteration median / min, measured 2026-10-03 04:13-04:16 UTC.

	baseline MAST3R_OPT=none (eth12, 1 CQ, float)	round start aff19e9 (eth12, 1 CQ, float)	now: eth12, 1 CQ, float	now: worker 12x10, 2 CQ, float	now: served path (worker 12x10, 2 CQ, uint8)
device forward (sum of the trace segments, synchronized)	62.22 / 62.10	40.01 / 39.81	39.64 / 39.37	39.78 / 39.44	39.78 / 39.51
host prep	0.86	0.94	0.98	0.88	0.46
H2D	1.07	1.04	1.03	1.07	0.60 (1.5 MB uint8)
D2H, synchronized, both maps incl. host assembly	29.00	3.34	3.09	3.09	3.01 (hidden in e2e)
e2e model(img1, img2)	92.15 / 91.89	45.06 / 44.44	44.57 / 44.06	43.22 / 42.72	42.01 / 41.84
real pair (kitchen, tools_prof/sym_check.py) e2e					42.03 / 41.54
return_pose (two maps + swapped pass), real pair e2e		84.14 / 83.73 (two-pass, today's graph)			60.41 / 60.07 (sym B-mode)
test_mast3r end_to_end (2 CQ, float)		44.79		42.82
eager profile: programs / kernel sum	1252 / 60.97	1230 / 38.07	1232 / 37.64

Accuracy, all gates pass:

	round start / float path now	uint8 served path now
synthetic randn pair head1 / head2 (all)	0.99846 / 0.99870 (0.99890); float outputs are bit-identical to the round start	n/a (randn is not a uint8 image)
synthetic smooth uint8 pair head1 / head2 (all)	float path on the same pixels 0.99667 / 0.99937 (0.99856); MAST3R_OPT=none 0.99752 / 0.99934 (0.99900)	0.99711 / 0.99929 (0.99855)
real pair pts3d PCC h1 / h2	0.99046 / 0.99081	0.99043 / 0.99091
real pair conf PCC h1 / h2	0.99182 / 0.99459	0.99220 / 0.99481
real pair median abs(dz)/z h1 / h2	1.13% / 1.18%	1.20% / 1.21%
real pair raw PCC vs half-pixel fp32 reference h1 / h2	0.999837 / 0.999849	0.999879 / 0.999884
test_mast3r end_to_end	0.9984 (h1 0.9984 / h2 0.9980) PASS

Honest notes:

• The uint8 path changes the arithmetic. The input is exact (v - 127.5), and 1/127.5 is folded into the patch-embed weight in fp64 with a single bf16 rounding. Against the half-pixel fp32 reference it is closer than the float path. Against the align-corners reference, its median relative depth error is 1.20 / 1.21% (float path 1.13 / 1.18%); the PCCs are equal or higher.
• On the smooth synthetic uint8 pair, every variant is below 0.998 on head 1, including the none baseline at 0.99752. The 0.998 gate is defined on the randn pair, where the float path is unchanged.
• The served-path e2e uses Tensix dispatch with 2 CQs. On the Galaxy that is 12x10, like ETH dispatch. On a real p150 that mode has 11x10 unless ETH dispatch supports 2 CQs there; the Galaxy ETH path has only 2 idle ETH cores, so it is 1 CQ. The like-for-like eth12 1-CQ float number is 44.57 / 44.06 ms.
• The host is shared by about 11 agents, and e2e medians moved by up to ±0.5 ms between runs in this round. The same-session A/B pairs are in the step table.

vs RTX 5090 (GPU_COMPARISON.md; the GPU "incl_h2d" row = upload of both views + forward + readback of both maps)

GPU variant	GPU ms	TT served path now (worker 2 CQ, uint8)	TT eth12 1 CQ float now
fp32 strict	103.4	42.0 (TT 2.5x faster)	44.6
tf32	70.8	42.0 (1.7x)	44.6
bf16 autocast	63.8	42.0 (1.5x)	44.6
fp16 autocast	56.6	42.0 (1.35x)	44.6
module converted to bf16	42.3	42.0 (TT 0.7% faster; real pair 42.03 / 41.54 min)	44.6 (GPU 5% faster)
module bf16 + SDPA	35.3	42.0 (GPU 1.19x faster)
bf16 module + torch.compile + CUDA graphs	21.1	42.0 (GPU 2x faster)
device-only (GPU excl_h2d bf16 module 41.3)	41.3	device 39.8 (TT faster)	39.6

The served-path TT number uses the server's uint8 views (1.5 MB upload) and readbacks that overlap compute on a second CQ. The GPU row uploads fp32 views; a GPU could also take uint8, which would save it about 0.5 ms of its 1.1 ms of transfers. So the bf16-module win is narrow (0.3 ms median), and it does not hold for the 1-CQ ETH configuration. Pose requests (return_pose) take 60.4 ms with the sym B-mode, against 2 x 42.3 = 84.6 ms for the GPU's two forwards. This is a mode-specific algorithmic saving (the encoder runs once and an unused head is skipped), so it is reported separately.

Round-2 steps (each kept step committed; A/B in one session where host noise matters)

commit	knob	change	effect
72b75eb	u8 (with hostcol)	uint8 pixel views. Host im2col is uint8 (1.5 MB instead of 3 MB). The device does typecast (exact) and v - 127.5 (exact in bf16). W/127.5 is folded into the patch-embed weight in fp64. The server preprocess returns uint8 HWC views (preprocess_image(..., uint8=True)), and warm-up uses a uint8 dummy	H2D 1.04 -> 0.58, host prep 0.94 -> 0.76 ms
72b75eb	MAST3R_CQS=2	forward captured as 2 trace segments (A: encoder, decoder, head 1; B: head 2). An event after A, then a CQ1 read of head 1 while CQ0 runs B	e2e 45.06 -> 43.95 (float) / 43.05 (u8); outputs bit-identical
167a332	u8	(py, px, c) im2col row order for uint8 (48-byte runs from the HWC array; weight rows permuted to match)	host_input 0.40 -> 0.18 ms, e2e 43.05 -> 42.68
3d850ff	phase	0/1 permutation matmuls at HiFi3. This is exact: the data operand sits in srcB and is fully covered by HiFi3, as probed with perm_mm_probe.py. Phase0 strip inputs are untilized into L1	bit-identical, trace 39.96 -> 39.88
2fa1785	ropes	RoPE trans_mat HEIGHT_SHARDED, one tile per core on all 120 cores (per-pass L1 copy), which selects the prefill-sharded rotary_embedding_llama factory. Probe: 26.7 -> 21.3 us (enc), 22.5 -> 18.0 us (dec)	bit-identical, trace 39.88 -> 39.49 ms (RoPE total 1.96 -> 1.61 ms)
440121c	sym (B-mode)	return_pose: one symmetric graph. The encoder runs once (it is per-view, so pass-2 inputs are the pass-1 encoder outputs), the decoder runs for both orders, then DPT head 1 + head 2 of (1,2) and head 1 of (2,1). Head 2 of (2,1) is skipped because PairViewer never reads it. Captured at server warm-up	out_ii / out_ji / out_jj bit-identical to the two-pass path; pose e2e 84.1 -> 60.4 ms; served forward 86 -> 60 ms (smoke test PASS, same pose / focals)
d087a52, f6cbac0	tsplit (needs 2 CQ)	4 segments: enc+dec+head-1 main rows, then head-2 main rows, then head-2 ring strips, then head-1 ring strips. Each 2 MB main-row readback overlaps later compute; only head 1's 128 KB strips are read after the last trace. Host assembly is incremental	bit-identical; e2e 42.49 / 42.58 -> 42.08 / 42.03 ms (same-session A/B)
f6a7dc5	serve	MAST3R_CQS=2 pinned in tt-model.yaml and SERVING.md	server smoke test PASS (forward 42-43 ms, pose 60 ms)

Round-2 tried and rejected

• RoPE math fidelity HiFi2 / HiFi3: no trace gain (RoPE is data-movement bound) and the outputs change. Reverted.
• cos/sin HEIGHT_SHARDED (with or without trans_mat), and RoPE with heads folded into batch: slower (rope_probe.py).
• Decoder fc1 with transposed-mcast 11x10 (110 cores) and encoder fc1 transposed: slower (99 vs 80 us, 214 vs 198 us; fc1_probe.py).
• MinimalMatmul K_block 2 for the decoder M=1024 linears and encoder proj (mm_sweep.py: 2-4 us faster per op in isolation): no trace gain in the model (39.58 vs 39.49 ms), synthetic head2 PCC 0.99862 vs 0.99870. Reverted.
• DPT 1x1 linears with explicit 1-D/2-D configs (lin1x1_probe.py): at most -28 us (128^2) and the outputs change; not taken.
• 128^2 refinenet relu/add in L1: conv circular buffers clash with the L1 tensors (TT_THROW, no hang). Reverted.
• Preallocated host buffers for D2H (copy_device_to_host_tensor): it does not write through to the torch buffer; net -0.03 ms. Not taken.
• glibc malloc tunables against page faults in the host assembly: no change.

Method (identical for every number below)

ROOT=/home/ttuser/experiments/tt-models; M=$ROOT/models/mast3r-p150; cd $M/code
source $ROOT/tools/chipenv.sh 9 $M/.venv >/dev/null
export TT_WEIGHTS_REVISION=61c57447d7b0adc8a1a30b2b0adec7a8935aa2a3
timeout -s INT 900 python bench_breakdown.py --mode eth12          # synthetic pair: PCC + per-phase timing (30 iters)
MAST3R_ETH=1 timeout -s INT 900 python tools_prof/real_pair_acc.py   # real pair accuracy gate
timeout -s INT 900 python test_mast3r.py --layer end_to_end --runs 10   # authors' gate (Tensix dispatch, 12x10)
MAST3R_TRACE=0 python -m tracy -r -p -v -o $TT_METAL_PROFILER_DIR/x --op-support-count 6000 tools_prof/prof_eager.py  # op profile

trace = execute_trace + synchronize_device of the whole forward (pure device time, one trace per forward); e2e = the public model(img1, img2) call (host prep + H2D + trace + D2H incl. host output assembly). The host is shared by ~11 agents, so host-side phases (host_prep, D2H, e2e) are noisy (D2H 2.4-4.1 ms run to run); the device trace is stable to about +-0.2 ms. Median and min are reported.

Accuracy gates (from OPT_BASELINE.md, kept): synthetic e2e PCC >= 0.998; real pair pts3d PCC >= 0.989 per head; conf PCC >= 0.99.

Round 1 results (chip 9, eth12, synthetic pair)

Final numbers measured 2026-10-03 02:45-02:52 UTC on HEAD f6b97f8 (two default runs, one MAST3R_OPT=none run in the same session).

	baseline (MAST3R_OPT=none)	round start (8dc072d)	now (f6b97f8)
trace (device forward) median / min	62.20 / 62.10 ms	46.47 / 46.23 ms	40.08 / 39.73 ms (2nd run 40.08 / 39.82)
host prep	0.83	0.84	0.88-0.98
H2D	1.06	1.03	1.03
D2H (incl. host output assembly)	28.00 (min 20.49)	2.49	3.13-3.49 (min 2.62)
e2e median / min	87.26 / 84.75 ms	50.48 / 50.13 ms	44.97-45.64 / 44.55 ms
test_mast3r end_to_end best-of-10 (Tensix dispatch 12x10)	84.49 ms	—	44.87 ms
p150-equivalent (--mode worker11, Tensix dispatch, 11x10) trace / e2e	62.62 / 89.16	—	41.41 / 47.07 ms
device programs per forward (eager profile)	1252	1212	1230
kernel-time sum (eager profile)	60.97 ms	44.91 ms	38.02 ms
synthetic PCC head1 / head2 (all)	0.99836 / 0.99895 (0.99909)	0.99845 / 0.99871	0.99846 / 0.99870 (0.99890)
test_mast3r end_to_end PCC (4 digits)	0.9984	0.9984	0.9984 (head1 0.9984 / head2 0.9980)
real pair pts3d PCC head1 / head2	0.98988 / 0.99076	0.99044 / 0.99079	0.99046 / 0.99081
real pair conf PCC head1 / head2	0.99237 / 0.99439	0.99178 / 0.99457	0.99182 / 0.99459
real pair raw PCC vs half-pixel-upsample fp32 ref	—	0.999839 / 0.999847	0.999837 / 0.999849
real pair median abs(dz)/z vs fp32 ref (vs half-pixel ref)	1.3%	1.05% / 1.11%	1.13% / 1.18% (0.37% / 0.33%)

All accuracy gates pass. Honest notes: the synthetic head2 PCC is 0.99870 vs 0.99895 at the baseline (head1 is higher, 0.99846 vs 0.99836); that shift came with the earlier sessions' gelut/lnfold steps, not this round. The real-pair median relative depth error moved 1.05/1.11% -> 1.13/1.18% within this round (phase/phase0 change the head arithmetic: folded kernels, no bf16 rounding of the upsampled activations); the PCCs are equal or higher.

Device: 62.2 -> 40.1 ms (-36%); this round 46.5 -> 40.1 ms (-14%). e2e: ~85-87 -> ~45 ms (-48%). Device time by stage now (eager profile, kernel sums): patch-embed + encoder 16.4 ms, decoder (incl. embed) 11.9 ms, DPT x2 9.8 ms; dispatch gaps ~2.0 ms (trace - kernel sum, ~1.6 us per program).

vs RTX 5090 (GPU_COMPARISON.md, same pair size, forward incl. H2D + readback of both maps)

GPU variant	GPU ms	TT now (e2e incl. H2D/D2H)	ratio TT/GPU
reference as shipped, fp32 strict	103.4	45.0	0.44 (TT faster)
tf32	70.8	45.0	0.64 (TT faster)
bf16 autocast	63.8	45.0	0.71 (TT faster)
fp16 autocast	56.6	45.0	0.80 (TT faster)
module converted to bf16	42.3	45.0	1.06 (GPU faster)
module bf16 + SDPA	35.3	45.0	1.27 (GPU faster)
bf16 module + torch.compile + CUDA graphs	21.1	45.0	2.13 (GPU faster)

The TT device forward alone (40.1 ms) is below the bf16-module GPU number but the x1-PCIe Galaxy transfers (H2D 1.0 + D2H ~3 ms incl. host assembly) put e2e just above it. The compiled GPU graph (21 ms) is out of reach without lower precision: LoFi matmuls were tried again this round per group (below) and fail the accuracy gates. Served-like e2e (PNG decode + preprocess + forward + npz encode, ~180 ms of identical host work on both sides) was not re-measured; it is host-bound on either accelerator (GPU 241-301 ms).

Kept steps (cumulative, each measured with the method above)

Earlier sessions (before this round; numbers from their commits):

commit	knob	change	trace
0e84074	out	compact channel-first head output, D2H 29 -> 3 ms	(e2e 89.7 -> 67.2)
c64edc6	mm	per-shape 12x10 matmul configs, exact GELU fused into fc1	62.5 -> 57.4
3fe76db	dpt	refinenet out_conv before the upsample, RM conv in/out	57.4 -> 55.9
4401161	l1	per-block transient activations in L1	55.9 -> 51.8
772e075	dpt	height-sharded refinenet convs >= 64x64	51.8 -> 50.3
13363fc/30c038f	dptf	HiFi2 (fp32 acc) DPT convs except the two head convs	50.3 -> 49.3
f7587b7	gelut	tanh-form GELU in the fc1 epilogue	49.3 -> 47.9
316339b	lnfold	LayerNorm affine folded into the following linear	47.9 -> 47.1
8dc072d	l1	per-pass L1 copies of the RoPE tables	47.1 -> 46.65

This round (re-validated HEAD 8dc072d first: trace 46.47, PCC 0.99845/0.99871, real pair gates pass):

commit	knob	change	trace	notes
206768d	phase	Polyphase DPT head tail. head.1-2 (bilinear x2, half-pixel, clamped == ttnn.upsample) + 3x3 conv == 4 phase 3x3 convs on the 256^2 grid with kernels K_ab = sum A[a] A[b] K (fp64 fold on host). Exact except the 2-px border ring, which is recomputed with the original upsample+conv on 2-row/2-col strips. The 512^2 x 128 upsample, its reshards and the 6-way DRAM-sliced conv are gone; head.4 (1x1, 128->4) is 4 tiny height-sharded linears summed.	46.47 -> 42.67	tail probe 3.21 -> 1.29 ms per head. D2H went up (host phase interleave)
ee66e6b	phase	phase interleave on device (RM permute + one exact 0/1 permutation matmul) -> NCHW rows; host only writes the ring	42.67 -> 43.20	D2H 4.5 -> ~2.7 ms, e2e -0.4..-2 ms
683d128	hostcol	host cast writes the patch-embed im2col layout [2, N, 768] bf16 directly (one strided copy, 0.14 ms, replaces cat+cast 0.97 ms); device drops reshape/permute im2col (0.7 ms)	43.20 -> 42.46	same 3 MB H2D
82bbe3e	phase	full-grid 2-D mcast config for the interleave matmul (was 16 cores, 123 us)	42.42 -> 42.22
c141100	lnfold	decoder: norm1(x)==norm_y(x) once the affine is folded, so each decoder step normalises x1 and x2 once for both branch blocks (-24 LN)	42.22 -> 41.93	bit-identical
07250e9	phase0	head.0 polyphase too: refinenet1's upsample folded into 4 phase convs at 128^2 (256->128); phases interleaved with an exact batched 0/1 matmul; final 6-px ring recomputed exactly by a thin-strip run of the original op sequence	41.93 -> 41.51	removes the 128->256 upsample + head.0's 3-way DRAM-sliced conv
ed59a76	phase	permutation matmuls at HiFi4 without fp32 acc (bit-exact; LoFi/HiFi2 are not, tools_prof/perm_mm_probe.py)	41.51 -> 41.45
8c74544	decb	decoder: the two branch blocks of a step share one B=2 split-heads / 2xRoPE / SDPA / concat-heads per attention (these ops cost ~the same at B=2 as at B=1: 28/21/12 vs 26/19/11 us); per-branch linears stay separate (a per-branch-weight bmm is 3-7x slower), outputs concatenated / sliced on batch	41.45 -> 40.25	same PCC
f6b97f8	resl1	encoder residual stream in L1 (patch-embed output copied to L1; residual-matmul outputs and their ones vector in L1; enc_norm output in DRAM)	40.25 -> 40.02	encoder LN 25.5 -> 20.0 us; same PCC

Exactness of the polyphase fold: host check (fp64) interior max abs diff 2e-8; device output vs fp32 torch: PCC 0.999993 for both the old and the new tail (tools_prof/phase_probe.py); real-pair raw PCC vs the half-pixel reference unchanged.

Tried and rejected (round 1)

• Row-sliced in-L1 tail (upsample + conv + 1x1 per row slice): correct but slower (3.5 vs 3.2 ms per head; the upsample itself was the cost).
• DPT resconv chain kept height-sharded in L1 (dpts): conv CBs of the 128^2x256 height-sharded convs (~1.2 MB, full weights per core) clash with any extra L1 tensor. Reverted.
• DPT relu/add in L1 interleaved for <= 64^2: -0.3 ms but changed PCC (conv picked a different config); not worth the accuracy churn. Reverted.
• Feeding the sharded upsample output straight into head.0's conv: CBs exceed L1 (2.4 MB). Superseded by phase0.
• Strip convs on fewer cores (16/32): no gain or CB clash.
• SDPA sweep on 12x10 (tools_prof/sdpa_probe2.py): q128/k256 HiFi2 (current) is the fastest; LoFi -2%, exp-approx 0%, fp32-acc +16%.
• LoFi for selected transformer matmul groups (experiment, not committed): decoder MLP only -0.25 ms but synthetic head1 PCC 0.99679; encoder MLP -1.0 ms, head1 0.99327; both MLPs -1.1 ms, head1 0.98269. All fail the 0.998 gate.
• Explicit minimal_matmul patch embed with L1 output (instead of linear + copy): bf16 packer accumulation over 6 K blocks changed outputs (head2 0.99852); with fp32 acc the PCC rose (0.99853/0.99889) but the real-pair depth error moved (1.29%/1.22%). Kept the bit-identical linear + copy.
• reallocate_halo_output=False for the DPT convs (drop the Move ops): no gain.
• GELU: the fused GELU_TANH epilogue costs ~100 us (enc fc1) / ~40 us (dec fc1) per call (3.4 ms per forward); no cheaper accurate SFPU variant exists in this tt-metal (the approximate GELU was rejected earlier for accuracy, PCC 0.993).

Megakernel / parameter taxonomy status

Round 2 changes:

• A (launch-fixed): MAST3R_CQS = 2 (trace segmentation + CQ1 readbacks), and the input dtype (uint8 in the server) is fixed at launch through hostcol.
• B (return_pose): now a precompiled second variant, the sym graph with 3-4 segments, captured at warm-up and selected per request. The other per-request fields remain C (pixel contents) or D (host only).
• Per forward: one H2D (1.5 MB uint8); trace segments replay back to back; each output is read on CQ1 as soon as its segment's event fires.

Round 1 text:

Per param_taxonomy/mast3r-p150.md: no request parameter changes device shapes (A: resolution 512, 1 pair, weights; C: pixel contents; B: return_pose = run the same trace twice). The whole forward is one metal trace with one H2D (3 MB im2col input) and two D2H (2 x 2.2 MB: NCHW maps + ring strips) per pair. Default knob set frozen as MAST3R_OPT=all (pinned in tt-model.yaml serve.env, documented in SERVING.md).

Remaining backlog (estimated device gain)

Round-2 view. Device time is now 39.6-39.8 ms; kernel sum 37.6 ms over 1232 programs.

• Phase0 ring strips: about 0.48 ms per head (two strip runs of about 25 small ops each). A cheaper exact border formulation, or asymmetric-padding convs that compute only the needed rows, could save about 0.3 ms in total.
• Tail phase-interleave chain: about 0.21 ms per head, incl. a 65 us RM reshape with a page-size change. An alternative layout for the 4 phase linears could remove the reshape and permute, about -0.2 ms in total.
• act_postprocess 1x1 + ConvTranspose fold (ap0 / ap1): about 190 us per head now. The fold is exact, but it needs a depth-to-space data movement (RM page-size change), so the net is uncertain, perhaps -0.2 ms.
• Decoder per-branch plumbing (concat 31 us/step, batch slices 14 us/step): blocked by per-branch weights. The bmm with per-branch weights is 3-7x slower.
• GELU_TANH epilogue (about 2.7 ms), SDPA (5.2 ms) and create-heads (1.4 ms) are at the stock-kernel floor at the gated precision.
• sym B-mode: branch-2 step 11 and dec_norm of the swapped pass are still computed (dead work, about 0.5 ms per pose request).

Round 1 list:

• Phase0 thin-strip pipeline: ~0.4 ms per head of small ops (halo/conv on tiny inputs); could be cut with a cheaper exact ring formulation.
• DPT refinenet plumbing at 64^2/128^2 (S2I/I2S per conv, DRAM relu/add): ~0.5 ms per head, blocked by conv L1 CB size.
• Dispatch gaps ~2 ms over ~1230 programs: fewer/fused small ops (DPT conv plumbing is ~4-5 programs per conv).
• GELU epilogue (3.4 ms) and SDPA (5.2 ms) are at the floor of the stock kernels at the gated precision; further device gains there need custom kernels (not done: tt-metal is frozen for this workflow).
• Decoder residual stream in L1: probe says ~1 us per LN, not worth it.
• models/tests/test_fused_host.py: 5 mock-graph tests fail since the earlier OPT commits (fake_ttnn lacks to_memory_config etc.); pre-existing, unchanged by this round.