GelSight image → normal force, in one page
Markerless gel, no F/T sensor, no training data from our rig. A physics pipeline with exactly one fitted number.
↖ results matrix overview action transform 中文The pipeline — six steps
nnmini): three-color illumination makes color→normal invertiblePost-processing: a 3-tap median over fresh tactile frames only (duplicated rows would let a row-wise filter count bad values three times). Cuts single-frame spikes from 4–8% to ≈0.
Photometric overhaul — classic per-sensor calibration (v2)
Ground-truth depth supervision (commanded press depth in GlowTact) exposed that the generic depth MLP recovers only ~25% of true indentation and saturates (peak-depth ρ = 0.39) — and, deeper, that the gsrobotics SDK's Poisson solver returns 39% of the amplitude even on a perfect synthetic gradient field (line-integral proved the gradients were correct at 105%). Rebuilt on the classic Dong/Yuan calibration: difference image → per-sensor RGB lookup table, self-calibrated from GlowTact's spherical presses via the exact relation a² = d(2R−d) (R = 3.35 mm, no external data) → exact Poisson (Dong's fast_poisson, 100.7% on the same benchmark) → sphere-supervised spatial gain field → Drake-style stiffening foundation p = k₁δ + k₂δ² with imprint-derived shape conditioning.
| stage (held-out, GlowTact 0–20 N) | ρ | MAE |
|---|---|---|
| MLP + linear Winkler (v1) | 0.63 | 4.4 N |
| LUT + solver fix + gain field + nonlinear foundation | 0.80 | 2.75 N |
| + imprint shape self-conditioning | 0.82 | 2.46 N |
| spheres only (geometry exact — the method's ceiling) | 0.91–0.94 | 1.5–1.9 N |
| spheres × 0–8 N (React's operating range; 7-seed median, +isotonic) | 0.95 (0.93–0.96) | 0.78 N (0.73–0.84) |
Ceiling context: the CNC's own commanded depth predicts force at ρ = 0.975. The remaining pooled gap is object-dependent contact mechanics; sub-newton MAE on 0–20 N exceeds what geometry alone carries (the 200K-frame supervised network reaches 2.1 N on the same range). A new sensor needs one 2-minute ball-press pass — a calibration the React rig can adopt.
Validation (v1 pipeline)
The v1 pipeline is validated on three force-labeled datasets (FEATS, FoTa cnc_Mini, GlowTact) and cross-checked against two neural estimators on identical frames — every predicted-vs-ground-truth scatter, per dataset, lives on the results page. Short version: physics 0.43-0.74 everywhere; each network 0.90+ in its own gel domain and collapsing outside it.
Optimizing against ground truth
The cnc_Mini force labels turned the pipeline's weak spots into measurable defects, fixed in order (each step verified on held-out data):
| step | evidence that drove it | ρ (held-out val) |
|---|---|---|
| baseline (volume, per-episode zeroing) | — | 0.34 pooled |
| + median zero map over scattered presses | only 4 of 2686 frames are truly contact-free; lowest-force references carried 1.7 N of baked-in contact | ≈ same (zero map wasn't the bottleneck) |
| + flat-field illumination normalization | force-fit residuals correlated with contact position (|ρ| up to 0.6); probe×quadrant conditioning raised ρ 0.45→0.55 — the vignette modulates the depth MLP's gain | 0.44 pooled |
| + edge filtering (contact centroid > 3 mm from border) | border presses sit where the vignette is steepest and imprints clip the sensor edge | 0.65 (probe-median 0.64) |
Volume or max depth? Settled empirically: the volume family wins (vol1.5-weighted 0.65, plain volume 0.63) over max depth (0.63) and clearly over contact area (0.35); a tiny 3-feature linear model matches ρ but halves MAE (0.61 N). The ceiling matters too: even the CNC's own commanded press depth only reaches ρ 0.78–0.88 within a probe — force at equal depth genuinely varies with texture and position.
Gallery
20 image samples (raw | indentation | 3D reconstruction | predicted vs ground-truth force) and 10 React episode clips (tactile | live depth | force trace): browse the full gallery.
NN vs model-based on the GelSight Mini — who has compared them?
No published head-to-head that we could find. The literature runs in two camps that cite but don't benchmark each other. Model-based: marker displacement × elasticity (Yuan 2017), photometric-stereo height + polynomial fit, inverse FEM (GelSlim, Ma 2019). Learned, on the Mini specifically: CANFnet (F/T-labeled, normal only), FEATS (FEA-labeled, 3D distributions), FeelAnyForce (200K ATI-labeled). Each motivates NN over physics qualitatively — FEA too slow for real time, linear elasticity misses elastomer nonlinearity — but their reported baselines are other networks, not the physics pipeline.
Our FEATS experiment is therefore one of the few direct data points: the model-based pipeline reaches ρ 0.70–0.85 with one fitted scalar and zero training frames, where the NNs earn sub-newton MAE in-domain but die outside their gel (FEATS on our markerless gel: no response at all). The trade is portability vs in-domain accuracy — and which one you need depends on whether you can collect labels on your own sensor.