Three estimators × four ground-truth datasets

The pattern is narrower than we first wrote it. FEATS collapses outside its own gel domain (0.96 → 0.04–0.07), and the physics pipeline does work on every dataset here (0.77–0.99) without ever seeing training data. But FeelAnyForce does not collapse: it holds 0.83 / 0.90 / 0.97 across the three markerless sets and only drops to 0.43 on the marker gel — and on Sparsh it beats us outright, 0.967 with zero labels against our 0.860 when our calibration is frozen to one pad. The honest claim is that the physics pipeline is the only estimator that needs no training data, not that it is the only one that transfers.

Re-run end to end after marker inpainting was added to the depth pipeline. One protocol on all four datasets: within each group (indenter family / probe / capture group / gel pad) half the frames fit a 5-feature least-squares model calibrated by isotonic regression, the other half are scored; 5 seeds, median reported. The control column repeats the identical protocol with the force labels permuted within each group — group structure and force distribution untouched, only the frame-to-force pairing destroyed.

Nothing moved. That is the expected result and it is worth being explicit about: marker inpainting was adopted for geometry only, so the force path was deliberately left byte-identical, and a spot check that recomputes cached features from the raw frames confirms it (max |cached − fresh| = 0 over 40 cnc frames). The last row is the same FEATS frames and the same splits with the marker-inpainted features fed to the force model instead — it loses 0.037 ρ, which is why it did not ship there.

A within-group shuffle is the right control here, not a global one: on FeelAnyForce the pooled ρ survived global shuffling at 0.442 vs 0.455, which is how we caught that its frame join had never been demonstrated. Reading the table: cnc and GlowTact sit ~0.9 above their controls; FEATS sits 0.78 above a control that is flat at 0.00.

FeelAnyForce, recovered

We previously excluded this dataset because its frame↔label join could not be demonstrated. The fix was not a better model but better data handling: instead of inferring a per-capture frame index, we range-extracted the original timestamped images from the publisher's split zip (5,202 members, 609 MB of an 82 GB archive, 5202/5202 CRC-32 verified). The image filename is the timestamp the label CSV references, so the join is filename-exact rather than inferred.

The control is now cleared by 0.62 where the inferred join cleared it by 0.013. Two further checks, because a jump from 0.455 to 0.961 invites the “too good” objection: join perturbation — re-labelling each frame with the force k frames later gives 0.603 at k=1 and 0.430 at k=25, which lands on the rejected inferred join's numbers, so the exact filename match is load-bearing and the old join behaved like one wrong by tens of frames; and a time-blocked split (fit on each capture's earliest half) still gives 0.912, so random half/half is not exploiting neighbouring frames. We also found the shipped CSVs put 3,188 frames in more than one official split (train/val/test), identical F_z on every duplicate — de-duplicated by path, otherwise one frame could land in both halves of our own split.

Datasets we evaluated but did not include

Four GelSight-Mini force datasets are rows above. Two more were worked on and left out, and it is worth saying why rather than leaving a reader to wonder.

Also checked and rejected for having no force ground truth in newtons, or the wrong sensor: Touch and Go, TVL, TacQuad, GelSLAM, 3DCal, DAR_OTS (GelSight model unstated; its marker-gel force labels are also a reordered copy of the markerless run), GenForce (marker images only, no RGB), AllSight (round finger, not a flat Mini gel), DIGIT and Wedge datasets. TacVerse is the one verified GelSight-Mini candidate still open — Mini with and without markers, 2 mm sphere, 0.1 mm steps — and it is access-gated.

In-domain, the FEATS U-net is excellent (ρ=0.96) — the negative results elsewhere are domain effects, not a weak model. FeelAnyForce, markerless-trained, degrades on the dotted gel (0.43): the same knife cuts both ways.

Removing the marker dots: a geometry win, not a force win

The dots occlude the gel, so the photometric table has no valid colour under them and Poisson integrates a dimple lattice into the depth map — visible as pockmarks all over the 3D mesh BEFORE panels. Detecting the dots on the reference and inpainting them out of both the reference and the frame before differencing (cv2 Telea, the image-space cousin of GelSight Wedge's Fig. 10 hole interpolation) removes it: lattice power at the 31.9 px marker pitch drops 1.523 → 0.890 (×0.65), lower on 91% of 120 frames above 1 N, Wilcoxon p = 2.6e-19. The detector is marker-specific, not a blob finder — 63/63 dots and 0 rejects on this reference, stable for every threshold from 3 to 16 grey levels, and exactly 0 blobs on the markerless GlowTact and cnc references, where the step is a bit-exact no-op.

Two things it does not do. It does not help force: on identical frames and splits ρ goes 0.7747 → 0.7371 and every paired median delta is negative, so the force features still come from the untouched pipeline. And it does not catch every dot — the dots shear with the gel (median 1.7 px, >8 px on 8% of frames), so a static reference mask leaves the displaced ones behind, which is what the residual dots in column 3 are. Controls: inpainting the same area of randomly placed fake markers gives 0.7697, i.e. no gain, so the small changes are not "inpainting = smoothing". What actually caps FEATS is the reference, not the dots — on the 20 lightest presses |dI| already sits at 11 grey levels off-dot and 88% of off-dot pixels pass the |dI|>8 valid test, so the mask is nearly the whole frame and the features integrate reference mismatch; a per-indenter light-press reference made it worse still (0.7747 → 0.7261).

Hard conditions: only 4 contact-free frames, 62% of presses near the pad border — the press grid is larger than the field of view (see pipeline debug). Strictly in view our ρ is 0.95 (MAE 0.25 N); FeelAnyForce reaches 0.92 non-edge.

Friendliest ground truth (centred presses, 10 free frames, 0–20 N). Per-indenter, calibrated within each family under the physical scope (contact fully in view, gel not bottomed out), ρ is 0.975–0.992 across all six indenters with MAE ≤ 0.73 N. Caveat: that calibration is refit per family, so it measures rank agreement within a group, not a transferable absolute-newton scale — see per-dataset calibration.

10 gel pads (6 sphere, 2 flat, 2 sharp), force in newtons. Our GlowTact table applied to this foreign sensor reaches ρ=0.878 on in-view frames. Rebuilding the table from Sparsh's own sphere presses — 708 frames, radius fitted at R=2.44 mm from a²=d(2R−d) — takes it to ρ=0.968, MAE 0.042 N, against a labels-shuffled-within-pad control of 0.23. Fitting on one pad and applying it unchanged to another costs nothing measurable (0.96–0.98 everywhere): one table, six gel pads.

Why the table matters more than the fit: with the wrong sensor's table a sphere press integrates to a bilobed shape with a central dip; with the self-calibrated table it is a single dome matching the analytic spherical cap (residual RMS 0.179 → 0.0545 mm). Measured before any integration, the LUT gradient sits 93.3° from the analytic sphere gradient — chance is 90° — and self-calibration brings it to 4.5° (within 30°: 15% → 99%). Reverse control: the Sparsh table fails on GlowTact frames too, so this is a per-sensor property, not a bad table.

What this costs and where it still fails. The price is one set of sphere presses with logged depth on the target sensor: this is calibrate once per sensor, not zero-shot. Rank order transfers across indenter shapes but absolute newtons do not — a sphere-fitted model applied to a flat punch degrades to MAE 0.37–0.40 N, and it got worse with the correct table, because true geometry widens the real feature-scale gap between a sphere and a punch. The sharp indenter is unsupported (in-view ρ 0.58). Shear stays out of reach by construction — the top shear decile keeps 1.6× the residual whichever table is used. Frames are restricted to a visible contact disc: 36% of presses show none and 11% are clipped, and the clipped subset carries the highest median force yet scores worse, so this is visibility, not force-range filtering. Three dataset defects had to be fixed first: flat/sharp trajectories ship 5 more frame indices than forces (silently drifting the labels, ρ≈0 until paired within each trajectory), sharp/batch_2 is stored BGR while the other nine are RGB, and flat/batch_2 ships only 3 of 4 image files.

Everything above scores force. Until now the depth underneath was only ever checked against our own analytic sphere cap, with the cap's amplitude anchored on the reconstruction itself. Tactile MNIST supplies what was missing: exact per-pixel depth, ray-cast from the 3D-printed digit meshes, on non-spherical geometry a sphere calibration cannot self-validate — 420 touches, 106 objects. The pose bookkeeping is verified end to end rather than assumed (re-rendering the ground-truth height map reproduces the shipped image to ~2/255 grey levels).

The finding is a range, not a number: accuracy is a steep function of press depth. Same digit meshes, re-rendered at five penetrations, no per-frame alignment and no fitted indentation scale:

At 0.3 mm the Type-2 error is 96.5 µm, below every one of 3D Cal's three published figures (152.8 / 171.6 / 290.0 µm) — and theirs are reported with a 2D cross-correlation alignment and a fitted indentation scale, ours with neither. At 0.6 mm (186.3 µm) we sit inside their range. At the 2.25 mm press this dataset actually ships we are an order of magnitude worse and recover barely half the peak. So: no accuracy number on this site should be quoted without the press depth it was measured at, and the working range of this reconstruction is shallow contact.

What it corrects, what it confirms.

One failure mode, now seen on four datasets. 420/420 of these touches have a contact that runs off the pad, and a control that moves a single sphere cap from mid-pad to the edge collapses its peak 1.39 → 0.30 mm against a 0.90 mm truth (Type-2 292 → 450 µm). That is the same effect as cnc_Mini's press grid being larger than the field of view (ρ 0.11 → 0.94 in the field-of-view ablation) and Sparsh's clipped-disc frames scoring worse despite carrying the highest median force. Contact visibility, not force range or gel type, is this pipeline's single biggest external failure mode — the Poisson solve's zero boundary cannot represent a surface that leaves the frame.

Honest caveat. These images are Taxim renders, not real GelSight frames, so this validates the geometry solver — table, mask, integration — rather than the sensor model, and the domain check above is what licenses reading it that way. Taxim's gel also follows the object geometry to within ~38 µm, so it barely models real gel non-conformance: the true numbers on a physical sensor at the same press depth will be worse, not better. Reproduce with python -m force_recovery.mnist_validation stage1|controls|sweep.

Sparsh is GelSight Mini without markers (the dataset card says so), ATI nano17 ground truth, loads to 3 N, three indenters. We had never run the baselines on it. Doing so on the identical frames — same pads, same in-view mask, our physics column reproducing ρ=0.9682 to the digit — gives an uncomfortable answer.

The headline is the unfitted row: both baselines are pretrained models that emit newtons and Sparsh's labels are newtons, so “reads force directly” and “correlates once re-calibrated per pad” are different claims. FeelAnyForce wins outright: 0.967 with zero labels against our 0.860 when we freeze the calibration to one pad (467 labels). Per-pad re-calibration hides that gap. It also needs no in-view filter — over all 7500 frames including clipped contacts it holds 0.897 where we fall to 0.683. On the sharp indenter, our weakest case, it is 0.878 against our 0.619.

FEATS's 0.384 is not signal. A pure-noise column scores 0.265 under the same protocol, and isotonic can flip its per-pad −0.099 to +0.261 by choosing a sign. Its predictions have an interquartile range of 0.002 N against the truth's 0.341 N — 93% of frames sit within 20% of its own median, and its MAE (0.341 N) is worse than predicting a constant (0.181 N). It is stuck, not wrong.

Two preprocessing facts that flip the result, recorded so nobody re-derives them: Sparsh's 320×240 frames are already gsdevice-cropped, so applying our 1/7 border crop again costs FeelAnyForce 0.985→0.976; and background subtraction is load-bearing — without it the model pins at 18.4–19.5 N and ρ goes negative.

The estimated normal force is written back into the dataset together with the action it implies: 36 episodes, 72 sensor-sides, 240,040 rows, 480,080 force samples. Row-count alignment verified 72/72, and a deliberately truncated file raises instead of silently truncating.

The pressing direction is not a coordinate axis: it comes from the rig's dual-ball calibration (pose-to-pose consistency ≤1.07°), and guessing “the tool z axis” would be 71–108° wrong. Verified independently of that calibration using kinematics alone: during force rise v·n̂>0 on 94.3% of sensor-sides, and corr(ΔF, v·n̂) is positive on 95.7%. Free space is exact identity — 219,518 no-contact rows have max|target − observed| = 0 element-wise, and the round trip k·‖target−observed‖ = F closes to 5.8e-14 N. Re-running the export reproduces byte-identical files.

1 N/mm is the declared starting point — and the full data says it is too soft

Stiffness is an assumption about the environment, not a measured property, so it is exported into the parquet field metadata and a sidecar beside the data rather than living only in code. Across all 480,080 samples it does not hold:

Read as a virtual impedance offset, 3–6 N/mm keeps the commanded target inside the gel; read as real gel compression, the estimator's own F / max depth gives a median of 15.4 N/mm (p5–p95 5.3–34.7). 1 N/mm is defensible only below about 4 N. An earlier version of this page inferred “0% past the gel” from a single sensor-side whose forces peaked at 2.3 N; the full-dataset number retired that claim.

On the dataset we actually care about, the two surviving estimators — physics (zero training) and FeelAnyForce (200K frames) — agree at ρ=0.91, and FeelAnyForce reads ≈0 N on every frame the physics pipeline calls contact-free. Neither can copy the other's mistakes.

For labelling React (markerless Mini): FeelAnyForce as the primary labeller, the physics pipeline as an independent audit, disagreement rows flagged. For any new gel or sensor where no trained model matches the domain: the physics pipeline is the only option that works out of the box — and its FEATS-dataset score (0.77) shows what it does on a domain nobody tuned it for.

And state the scope with it. Those ρ are force scores, per group. The depth underneath is now measured against exact per-pixel ground truth and is a strong function of press depth — 11 µm MAE at 0.3 mm, 281 µm at 2.25 mm — so the honest one-line summary is: accurate shallow-contact geometry, monotone force within a calibrated group, and neither claim survives a contact that leaves the field of view. Marker gels get one extra step (dots inpainted before differencing) which buys geometry and not force.