GelSight image → normal force, in one page

Ground-truth depth supervision (commanded press depth in GlowTact) exposed that the generic depth MLP recovers only ~25% of true indentation and saturates (peak-depth ρ = 0.39) — and, deeper, that the gsrobotics SDK's Poisson solver returns 39% of the amplitude even on a perfect synthetic gradient field (line-integral proved the gradients were correct at 105%). Rebuilt on the classic Dong/Yuan calibration: difference image → per-sensor RGB lookup table, self-calibrated from GlowTact's spherical presses via the exact relation a² = d(2R−d) (R = 3.35 mm, no external data) → exact Poisson (Dong's fast_poisson, 100.7% on the same benchmark) → sphere-supervised spatial gain field → Drake-style stiffening foundation p = k₁δ + k₂δ² with imprint-derived shape conditioning.

Ceiling context: the CNC's own commanded depth predicts force at ρ = 0.975. The remaining pooled gap is object-dependent contact mechanics; sub-newton MAE on 0–20 N exceeds what geometry alone carries (the 200K-frame supervised network reaches 2.1 N on the same range). A new sensor needs one 2-minute ball-press pass — a calibration the React rig can adopt.

The LUT-v2 pipeline is validated on four force-labeled datasets (FEATS, FoTa cnc_Mini, GlowTact, Sparsh) and cross-checked against two neural estimators on identical frames — every predicted-vs-ground-truth scatter, per dataset, lives on the results page. Short version: physics 0.77-0.99 everywhere (GlowTact 0.99, cnc_Mini in-view 0.95, Sparsh in-view 0.97, FEATS 0.77), each reported beside a within-group label-shuffle control that lands between −0.00 and 0.26; each network 0.90+ in its own gel domain and collapsing outside it.

Every ρ above scores force. The depth underneath used to be checked only against our own analytic sphere cap. It is now measured against exact per-pixel ground truth — ray-cast mesh depth on 420 Tactile MNIST touches, non-spherical geometry a sphere calibration cannot self-validate — and the answer is a range set by press depth, not a number:

At 0.3 mm, with no per-frame alignment and no fitted indentation scale, the Type-2 error is 96.5 µm — below all three of 3D Cal's published figures (152.8 / 171.6 / 290.0 µm), which are reported with both. By 2.25 mm we recover barely half the peak. The working range of this reconstruction is shallow contact, and no accuracy number here should be quoted without its press depth.

The same run corrected two of our own claims and confirmed a third: flat-top over-doming is +7% to +12%, not the +23–42% we implied (compliant gel wraps a flat edge, so the true centre/rim ratio is 1.33–1.40, not 1.0); unobserved LUT bins are a minor factor (correlation 0.10–0.29 with error); and the |dI|>8 valid mask really is halo-dominated (IoU 0.614, recall 0.917, over-segmentation 0.531). The photometric table is not the weak link — its gradients sit 24.4° from the true ones off-domain versus 26.1° on its own sensor. Full tables, controls and the Taxim caveat on the results page.

FEATS is our only dotted gel. The dots occlude the gel, so the photometric table has no valid colour underneath them and Poisson integration bakes a dimple lattice into the depth map — visible as pockmarks across the whole reconstructed surface. The fix is the one GelSight Wedge (Wang/She/Dong/Adelson, ICRA 2021) gives in its Fig. 10 for marker holes — detect the dots, then fill them rather than mask them — moved from gradient space into image space: inpaint the dots out of both the reference and the frame (cv2 Telea) and difference the inpainted pair.

Two things worth reading off that table. Filling beats masking: setting g := 0 on the holes barely helps (×0.93) because a zero patch puts a dipole layer on every hole boundary, at exactly the lattice frequency it is supposed to remove. And the geometry column and the force column disagree — the variant that halves the dimple power is the one that loses the most ρ. So the step ships for depth and 3D only (marker_removal.stages_depth, which wraps rather than edits the force path); the force features on FEATS still come from the untouched pipeline. Everything else on this site is markerless, where the detector finds zero dots and the step is a bit-exact no-op.

Controls, because “inpainting helps” is an easy thing to fool yourself about: inpainting the same area of randomly placed fake markers gives ρ 0.7697 — no gain, so this is not smoothing. The detector finds 63/63 dots with 0 rejects on the FEATS reference and stays at 63 for every threshold from 3 to 16 grey levels, but exactly 0 blobs on the GlowTact and cnc references, so it is marker-specific. It is also not a complete fix: the dots shear with the gel (median 1.7 px, >8 px on 8% of frames) and a static reference mask leaves the displaced ones in place. Full study: python -m force_recovery.marker_study all.

Worth stating plainly, because it bounds what those ρ mean: the pipeline holds no hardness or elastic-modulus constant anywhere. Nothing in the code knows a gel's Shore hardness or Young's modulus. Stiffness enters only implicitly — absorbed into the per-group least-squares weights and the isotonic calibration stacked on them — and those are refit per dataset and per indenter/probe group: every cnc_Mini probe, every GlowTact indenter family, every FEATS capture group gets its own fit, half the group fitting and half held out.

The only physical constants shared across datasets are the sphere-calibrated RGB lookup table and MM_PER_PIXEL. Everything downstream of depth is refit. The reported ρ are therefore per-group rank correlations — evidence that the geometry recovered from the image is monotone in force within a group. They do not demonstrate a transferable absolute-newton model across gels; the one pooled multi-object fit we tried was much worse (ρ 0.47), which is why the sphere, with known geometry, is the calibration object.

The reconstruction underneath is not the noise source it appears to be, and that is measured, not assumed: high-frequency content in the depth field is 0.3% of peak depth (6.5 µm on a 1.9 mm press), while in the gradient field it is 22.2% — LUT bin quantisation, with 9.8% of contact pixels landing in bins the sphere calibration never observed (nearest-filled). Poisson integration is a low-pass, so the LUT noise is gone by the time depth exists; the speckle visible in gradient-domain debug panels never reaches the force features.

The cnc_Mini force labels turned the pipeline's weak spots into measurable defects, fixed in order (each step verified on held-out data):

Volume or max depth? Settled empirically: the volume family wins (vol^1.5-weighted 0.65, plain volume 0.63) over max depth (0.63) and clearly over contact area (0.35); a tiny 3-feature linear model matches ρ but halves MAE (0.61 N). The ceiling matters too: even the CNC's own commanded press depth only reaches ρ 0.78–0.88 within a probe — force at equal depth genuinely varies with texture and position.

20 image samples (raw | indentation | Open3D mesh of the reconstruction | predicted vs ground-truth force) and 10 React episode clips (tactile | live depth | live Open3D mesh | force trace): browse the full gallery. The mesh panel re-renders on every fresh tactile frame.

No published head-to-head that we could find. The literature runs in two camps that cite but don't benchmark each other. Model-based: marker displacement × elasticity (Yuan 2017), photometric-stereo height + polynomial fit, inverse FEM (GelSlim, Ma 2019). Learned, on the Mini specifically: CANFnet (F/T-labeled, normal only), FEATS (FEA-labeled, 3D distributions), FeelAnyForce (200K ATI-labeled). Each motivates NN over physics qualitatively — FEA too slow for real time, linear elasticity misses elastomer nonlinearity — but their reported baselines are other networks, not the physics pipeline.

Our FEATS experiment is therefore one of the few direct data points: the model-based pipeline reaches ρ 0.70–0.85 with one fitted scalar and zero training frames, where the NNs earn sub-newton MAE in-domain but die outside their gel (FEATS on our markerless gel: no response at all). The trade is portability vs in-domain accuracy — and which one you need depends on whether you can collect labels on your own sensor.

Five candidate improvements, each with a control that could have killed it. Two survived. Reproduce with python -m force_recovery.improvement_study all.

The one real structural finding: a single monotone volume feature generalises to unseen indenter shapes better than the five-feature model, while five features stay better once the shape is calibrated. On GlowTact leave-one-indenter-out — the model never sees the held-out shape — this repairs exactly the two weak spots we reported earlier:

Every held-out indenter now lands at ρ ≥ 0.986 without the model ever seeing that shape — a stronger claim than the per-indenter calibrated numbers on the results page, which are fitted within each family. The cost is absolute newtons: five features still win MAE when the shape is known (Sparsh cross-pad 0.0372 vs 0.0443 N), so the rule is volume-only for unseen shapes, five features once calibrated. A learned g inside the physics law collapses entirely (ρ −0.249): isotonic clips monotonically outside the training range, a network extrapolates freely.