Recovering Force Actions for the React Dataset

What was tried first — and why it failed

FEATS transfer: a negative result

The plan matched FARM (arXiv:2510.13324): run FEATS (U-net, GelSight-Mini images → FEA-supervised force distributions, MAE < 1 N) offline over the recordings. It does not transfer: FEATS is trained on marker-dot gel, and the React sensors use markerless gel — different texture, different illumination profile.

Left: FEATS training sensor (63 marker dots). Right: React sensor, markerless. On this input the network returns its no-contact output for every frame — the strongest-contact frame of pushT/episode_000 predicts −0.040 N total, identical to a free frame (−0.042 N). Dot-based per-sensor calibration is impossible for the same reason.

Method 1 — Photometric-stereo depth → Winkler normal force

GelSight frame
640×480 RGB→ per-pixel MLP
(gsrobotics, markerless)→ surface normals→ Poisson (DCT)
integration→ background plane
removal (robust fit)→ indentation δ(x,y) [mm]→ Winkler foundation
F = c · Σδ · dA

The gel pad is modelled as a bed of independent springs (thin-elastic-layer approximation); the constant c is fitted on FEATS ground truth rather than assumed (see the external-validation section). Per-sensor calibration uses the 15 lowest-intensity frames of each episode: median depth becomes the zero map and the MAD of the plane-removed residuals sets a 5σ contact threshold — a fixed threshold cannot work because reconstruction noise differs per sensor (measured ~10 µm after plane removal). Each frame additionally gets a robust background-plane fit, which removes the frame-to-frame illumination tilt that otherwise dominates integrated depth.

Strongest contact of motherboard/episode_000 (left sensor): raw frame | difference vs reference | reconstructed depth. Indentation localises exactly where the image difference shows contact.

Evaluation (no ground truth → falsifiable properties instead)

E1.1 specificity — no-contact rows must read ≈0 N; SNR = median force on high-intensity rows / p95 on reference rows. E1.2 correlation — Spearman ρ against contact intensity, an independent statistic of the same images. E1.3 spikes — single-frame excursions are solver noise; a median-3 over fresh frames must remove them. E1.4 calibration stability across episodes.

episode	side	fresh rows	max F [N]	SNR	ρ vs intensity	spike raw→filt [%]	threshold [µm]
2026-05-10/episode_000	left	2010	7.56	2675186.2	0.92	1.4 → 0.0	10
2026-05-10/episode_000	right	2007	7.27	9841.9	0.83	1.0 → 0.0	10
2026-05-10/episode_001	left	2157	9.39	646.7	0.84	1.4 → 0.0	10
2026-05-10/episode_001	right	2155	8.30	1373956.4	0.76	1.1 → 0.0	10
2026-05-10/episode_002	left	3037	9.66	623.9	0.78	1.0 → 0.0	10
2026-05-10/episode_002	right	3025	15.48	1977041.6	0.80	1.6 → 0.0	10
2026-05-10/episode_003	left	2629	5.00	941.3	0.82	0.1 → 0.0	10
2026-05-10/episode_003	right	2629	3.10	2.0	0.81	0.5 → 0.0	10
2026-05-10/episode_004	left	690	3.28	0.7	0.70	0.1 → 0.0	10
2026-05-10/episode_004	right	692	4.77	2.0	0.89	0.3 → 0.0	10
2026-05-10/episode_005	left	220	2.87	1.5	0.82	0.0 → 0.0	10
2026-05-10/episode_005	right	221	3.55	16.6	0.74	0.0 → 0.0	10
2026-05-10/episode_006	left	1153	3.11	72398.3	0.89	0.1 → 0.0	10
2026-05-10/episode_006	right	1152	4.18	1627990.2	0.94	0.0 → 0.0	10
2026-05-10/episode_007	left	556	2.10	0.8	0.68	0.2 → 0.0	10
2026-05-10/episode_007	right	556	3.21	1371906.6	0.90	0.2 → 0.0	10
2026-05-10/episode_008	left	376	2.71	309.3	0.93	0.0 → 0.0	10
2026-05-10/episode_008	right	377	2.09	3.3	0.82	0.3 → 0.0	10
2026-05-10/episode_009	left	523	4.53	119.1	0.88	0.6 → 0.0	10
2026-05-10/episode_009	right	526	5.42	166207.3	0.92	1.1 → 0.0	10
2026-05-10/episode_010	left	300	4.12	5.2	0.91	0.7 → 0.0	10
2026-05-10/episode_010	right	299	2.38	3.3	0.73	2.0 → 0.0	10
2026-05-10/episode_011	left	635	8.89	70.4	0.88	0.0 → 0.0	10
2026-05-10/episode_011	right	636	2.16	2668.3	0.88	0.0 → 0.0	10
2026-05-11/episode_003	left	2796	9.49	42.5	0.78	1.7 → 0.0	10
2026-05-11/episode_003	right	2804	17.32	2880060.2	0.78	2.4 → 0.0	10
2026-05-11/episode_004	left	735	10.49	591.4	0.76	1.5 → 0.0	10
2026-05-11/episode_004	right	736	10.42	2597437.6	0.78	1.0 → 0.0	10
2026-05-11/episode_005	left	4038	9.20	8449.8	0.84	1.0 → 0.0	10
2026-05-11/episode_005	right	4036	11.16	3092505.9	0.82	1.1 → 0.1	10
2026-05-11/episode_006	left	3361	10.78	17.8	0.85	0.8 → 0.0	10
2026-05-11/episode_006	right	3364	14.07	4450209.1	0.83	1.1 → 0.0	10
2026-05-11/episode_007	left	1923	8.64	2166.5	0.82	0.4 → 0.0	10
2026-05-11/episode_007	right	1923	8.67	1010.9	0.77	1.1 → 0.1	10
2026-05-11/episode_008	left	3827	12.97	12594.2	0.78	0.9 → 0.0	10
2026-05-11/episode_008	right	3824	13.40	4091582.3	0.85	1.4 → 0.0	10
2026-05-11/episode_009	left	1213	8.05	723.4	0.84	0.7 → 0.0	10
2026-05-11/episode_009	right	1212	14.28	4749535.1	0.91	1.3 → 0.0	10
2026-05-11/episode_010	left	1903	4.21	691.1	0.71	0.4 → 0.0	10
2026-05-11/episode_010	right	1896	17.67	4123979.6	0.87	1.7 → 0.1	10
2026-05-11/episode_011	left	2133	7.94	15.5	0.82	0.8 → 0.0	10
2026-05-11/episode_011	right	2127	10.83	3515466.9	0.87	2.0 → 0.1	10
2026-05-11/episode_012	left	4131	10.33	514.8	0.89	1.4 → 0.0	10
2026-05-11/episode_012	right	4130	12.99	4057872.8	0.88	1.8 → 0.0	10
2026-05-11/episode_013	left	3499	1.87	2.3	0.72	0.2 → 0.0	10
2026-05-11/episode_013	right	3508	2.51	716545.8	0.83	0.8 → 0.1	10
2026-05-11/episode_014	left	675	1.45	664.6	0.88	0.3 → 0.0	10
2026-05-11/episode_014	right	679	2.53	370.0	0.86	0.4 → 0.0	10
2026-05-11/episode_015	left	1650	1.73	9284.3	0.87	0.3 → 0.0	10
2026-05-11/episode_015	right	1647	4.70	1556752.2	0.88	2.9 → 0.0	10
2026-05-11/episode_016	left	3269	2.04	57.1	0.84	0.8 → 0.0	10
2026-05-11/episode_016	right	3252	4.40	1357047.6	0.82	2.0 → 0.0	10
2026-05-11/episode_017	left	3980	20.57	1407.6	0.83	1.3 → 0.0	10
2026-05-11/episode_017	right	3954	22.80	65.7	0.89	1.5 → 0.0	12
2026-05-19/episode_000	left	518	8.69	9.3	0.93	2.7 → 0.2	10
2026-05-19/episode_000	right	506	11.12	6884184.8	0.94	4.0 → 0.2	10
2026-05-19/episode_001	left	251	8.23	1.6	0.89	4.0 → 0.4	24
2026-05-19/episode_001	right	246	9.45	1.5	0.93	2.5 → 0.0	10
2026-05-19/episode_002	left	590	6.95	975.9	0.94	4.6 → 0.0	10
2026-05-19/episode_002	right	594	9.68	22.1	0.92	5.4 → 0.2	10
2026-05-19/episode_003	left	29	5.04	1.1	0.18	3.7 → 0.0	30
2026-05-19/episode_003	right	31	4.98	4836304.2	0.93	6.9 → 0.0	70
2026-05-19/episode_004	left	35	4.14	1.1	0.80	6.1 → 0.0	23
2026-05-19/episode_004	right	41	6.08	1.7	0.95	2.6 → 0.0	10
2026-06-18/episode_000	left	689	1.97	164912.1	0.77	0.6 → 0.1	10
2026-06-18/episode_000	right	693	3.70	19362.2	0.87	1.4 → 0.0	10
2026-06-18/episode_001	left	4099	7.23	317003.9	0.80	0.5 → 0.0	10
2026-06-18/episode_001	right	4153	6.65	575866.7	0.79	0.7 → 0.0	10
2026-06-18/episode_002	left	3956	4.52	337766.3	0.80	0.4 → 0.0	10
2026-06-18/episode_002	right	3994	5.59	793320.5	0.84	1.2 → 0.0	10
2026-06-18/episode_003	left	4021	3.52	209622.6	0.77	0.1 → 0.0	10
2026-06-18/episode_003	right	4065	4.98	628000.0	0.83	0.7 → 0.0	10

12 s around each episode's force peak: live force bar under the tactile stream (median-3 filtered). Left: pushT (light, intermittent contact). Right: motherboard (sustained presses).

External validation — FEATS ground truth

React has no force ground truth, but FEATS (arXiv:2411.03315) does: a GelSight Mini pressed by a CNC machine against 24 indenters, with FEA-derived force labels. Running our estimator on its val split is a transfer test — marker-dot gel, different pad, single global no-contact reference — i.e. strictly harder than the React setting, where reference frames come from the same episode seconds away.

metric	value	scope
Spearman ρ (pooled)	0.70	normal loading, ≤30 N, n=150
Pearson r (pooled)	0.63
MAE after scale calibration	4.7 N
ρ within one indenter family	0.66–0.94	zelda 0.94, cylinder 0.75, triangle 0.70
ρ on unseen indenter shapes	0.85	test_unknown_indenters split, n=100 — shape generalization holds
scale stability across splits	2–4×	fitted N/mm³ varies 1.9–7.2 between sessions/sensors: within-episode relative force is reliable, cross-sensor absolute scale is not
ρ on shear-loaded captures	−0.15	excluded & documented: shear churns the image without adding indentation volume
fitted scale	1.89 N/mm³	40× the assumed Winkler constant — the fit absorbs the depth-unit and gel-constant assumptions; React forces are reported in FEATS-calibrated newtons

What the validation bought. Chasing the initial ρ = 0.42 exposed two pipeline defects that also affected React inference: a global illumination tilt dominating the integrated depth (fixed by a robust background-plane fit per frame — React contact thresholds tightened from 50 µm to 10 µm) and marker-dot imprints from the SDK's mask-and-interpolate path (fixed by inpainting the dots before the network). A quadratic background model was also tried and rejected — it absorbed real contact (ρ 0.61→0.50). Sanity check on scale: the strongest motherboard press reads ~0.2 N under the theoretical constant but ~7 N calibrated — the latter is what pressing a connector home actually takes.

Method 2 — DexForce-style force-informed position targets

Rather than adding force as a new action dimension, the estimated normal force becomes a virtual position target past the contact surface (DexForce, arXiv:2501.10356):

p_target = p_observed + (F̂_n / k) · n̂, k = 1500 N/m (deployment impedance)

The action stays a pose — it composes with the existing 30 Hz pose actions, needs only an impedance controller at deployment, and in free space reduces exactly to the observed pose. The pressing direction n̂ is not a guessed axis: it is the rig's dual-ball calibrated gel_axis_in_rigid (pose-to-pose consistency ≈1°), rotated into world frame per row. The naive [0,0,1] guess produced negative approach alignment at force onsets; the calibrated axis makes it positive.

Evaluation

E2.1 free-space invariance — zero force must leave the action exactly the observed pose. E2.2 boundedness — F/k must stay millimetre-scale to be a safe action. E2.3 roundtrip — k·‖target − pose‖ must reproduce the input force (pure algebra). E2.4 geometry — motion should align more with n̂ while pressing than in free motion.

episode	side	free-space max offset [m]	penetration p50 / max [mm]	roundtrip err [N]	v·n̂ pressing / free
2026-05-10/episode_000	left	0.0e+00	1.10 / 5.04	8.2e-14	+0.128 / -0.005
2026-05-10/episode_000	right	0.0e+00	0.97 / 4.85	6.1e-14	+0.037 / +0.025
2026-05-10/episode_001	left	0.0e+00	1.33 / 6.26	6.4e-14	+0.145 / +0.079
2026-05-10/episode_001	right	0.0e+00	1.12 / 5.54	6.0e-14	+0.221 / +0.082
2026-05-10/episode_002	left	0.0e+00	0.95 / 6.44	5.2e-14	+0.163 / +0.075
2026-05-10/episode_002	right	0.0e+00	1.16 / 10.32	7.0e-14	+0.075 / +0.081
2026-05-10/episode_003	left	0.0e+00	0.61 / 3.33	3.6e-14	-0.024 / +0.072
2026-05-10/episode_003	right	0.0e+00	0.39 / 2.06	6.9e-14	+0.026 / +0.000
2026-05-10/episode_004	left	0.0e+00	0.71 / 2.19	2.5e-14	-0.017 / +0.041
2026-05-10/episode_004	right	0.0e+00	0.67 / 3.18	4.1e-14	-0.031 / -0.164
2026-05-10/episode_005	left	0.0e+00	1.01 / 1.91	1.8e-14	-0.059 / -0.020
2026-05-10/episode_005	right	0.0e+00	0.74 / 2.37	2.9e-14	-0.033 / +0.007
2026-05-10/episode_006	left	0.0e+00	0.69 / 2.07	2.6e-14	-0.027 / +0.101
2026-05-10/episode_006	right	0.0e+00	0.55 / 2.79	3.2e-14	+0.034 / -0.009
2026-05-10/episode_007	left	0.0e+00	0.55 / 1.40	3.9e-14	-0.010 / +0.188
2026-05-10/episode_007	right	0.0e+00	0.72 / 2.14	2.8e-14	-0.004 / -0.053
2026-05-10/episode_008	left	0.0e+00	0.37 / 1.81	3.6e-14	-0.021 / +0.205
2026-05-10/episode_008	right	0.0e+00	0.56 / 1.39	2.6e-14	+0.047 / -0.053
2026-05-10/episode_009	left	0.0e+00	0.85 / 3.02	4.1e-14	-0.062 / +0.103
2026-05-10/episode_009	right	0.0e+00	1.36 / 3.61	4.8e-14	+0.012 / +0.066
2026-05-10/episode_010	left	0.0e+00	0.72 / 2.75	3.0e-14	-0.031 / +0.117
2026-05-10/episode_010	right	0.0e+00	0.86 / 1.59	2.9e-14	+0.061 / +0.082
2026-05-10/episode_011	left	0.0e+00	0.47 / 5.93	3.5e-14	-0.014 / +0.078
2026-05-10/episode_011	right	0.0e+00	0.58 / 1.44	4.7e-14	+0.004 / +0.023
2026-05-11/episode_003	left	0.0e+00	1.34 / 6.33	4.5e-14	+0.115 / +0.102
2026-05-11/episode_003	right	0.0e+00	1.73 / 11.55	6.8e-14	+0.079 / +0.073
2026-05-11/episode_004	left	0.0e+00	2.38 / 6.99	4.5e-14	+0.146 / +0.127
2026-05-11/episode_004	right	0.0e+00	1.83 / 6.95	6.6e-14	+0.079 / +0.105
2026-05-11/episode_005	left	0.0e+00	1.70 / 6.13	7.3e-14	+0.075 / +0.022
2026-05-11/episode_005	right	0.0e+00	1.62 / 7.44	8.0e-14	+0.036 / -0.011
2026-05-11/episode_006	left	0.0e+00	1.95 / 7.19	4.2e-14	+0.083 / +0.110
2026-05-11/episode_006	right	0.0e+00	2.16 / 9.38	8.1e-14	+0.070 / +0.115
2026-05-11/episode_007	left	0.0e+00	1.56 / 5.76	3.3e-14	+0.007 / +0.122
2026-05-11/episode_007	right	0.0e+00	1.16 / 5.78	7.3e-14	+0.040 / +0.096
2026-05-11/episode_008	left	0.0e+00	1.95 / 8.65	3.6e-14	+0.051 / +0.124
2026-05-11/episode_008	right	0.0e+00	2.05 / 8.93	6.5e-14	+0.051 / +0.084
2026-05-11/episode_009	left	0.0e+00	0.82 / 5.37	3.2e-14	+0.048 / +0.065
2026-05-11/episode_009	right	0.0e+00	1.86 / 9.52	4.3e-14	-0.015 / +0.072
2026-05-11/episode_010	left	0.0e+00	0.74 / 2.81	4.6e-14	+0.140 / +0.068
2026-05-11/episode_010	right	0.0e+00	1.95 / 11.78	5.8e-14	+0.074 / +0.009
2026-05-11/episode_011	left	0.0e+00	1.03 / 5.29	3.4e-14	+0.147 / +0.094
2026-05-11/episode_011	right	0.0e+00	1.81 / 7.22	4.0e-14	+0.060 / +0.008
2026-05-11/episode_012	left	0.0e+00	1.02 / 6.89	4.2e-14	+0.086 / +0.092
2026-05-11/episode_012	right	0.0e+00	1.72 / 8.66	6.2e-14	+0.016 / +0.036
2026-05-11/episode_013	left	0.0e+00	0.25 / 1.25	5.0e-14	+0.005 / +0.057
2026-05-11/episode_013	right	0.0e+00	0.38 / 1.67	7.0e-14	+0.058 / +0.081
2026-05-11/episode_014	left	0.0e+00	0.22 / 0.97	3.7e-14	-0.015 / +0.046
2026-05-11/episode_014	right	0.0e+00	0.30 / 1.69	6.6e-14	+0.073 / +0.066
2026-05-11/episode_015	left	0.0e+00	0.27 / 1.16	7.2e-14	-0.018 / +0.041
2026-05-11/episode_015	right	0.0e+00	0.71 / 3.14	3.5e-14	+0.067 / +0.111
2026-05-11/episode_016	left	0.0e+00	0.30 / 1.36	4.0e-14	+0.017 / +0.074
2026-05-11/episode_016	right	0.0e+00	0.62 / 2.93	5.4e-14	+0.012 / +0.124
2026-05-11/episode_017	left	0.0e+00	3.94 / 13.72	5.0e-14	+0.044 / +0.156
2026-05-11/episode_017	right	0.0e+00	4.22 / 15.20	7.5e-14	-0.004 / +0.126
2026-05-19/episode_000	left	0.0e+00	1.51 / 5.79	3.0e-14	+0.021 / -0.011
2026-05-19/episode_000	right	0.0e+00	3.22 / 7.41	3.6e-14	+0.036 / -0.018
2026-05-19/episode_001	left	0.0e+00	1.56 / 5.48	3.5e-14	+0.004 / +0.133
2026-05-19/episode_001	right	0.0e+00	2.97 / 6.30	2.4e-14	+0.050 / +0.007
2026-05-19/episode_002	left	0.0e+00	1.69 / 4.63	2.9e-14	+0.046 / -0.069
2026-05-19/episode_002	right	0.0e+00	1.78 / 6.45	3.5e-14	+0.054 / +0.008
2026-05-19/episode_003	left	0.0e+00	1.50 / 3.36	1.8e-14	-0.045 / +nan
2026-05-19/episode_003	right	0.0e+00	2.23 / 3.32	2.2e-14	+0.023 / +0.106
2026-05-19/episode_004	left	0.0e+00	2.00 / 2.76	2.1e-14	+0.004 / +0.397
2026-05-19/episode_004	right	0.0e+00	2.81 / 4.05	3.8e-14	-0.213 / +0.278
2026-06-18/episode_000	left	0.0e+00	0.28 / 1.32	4.2e-14	-0.067 / +0.073
2026-06-18/episode_000	right	0.0e+00	0.44 / 2.47	6.5e-14	+0.080 / -0.003
2026-06-18/episode_001	left	0.0e+00	0.30 / 4.82	4.5e-14	+0.050 / +0.087
2026-06-18/episode_001	right	0.0e+00	0.43 / 4.43	9.1e-14	+0.066 / +0.051
2026-06-18/episode_002	left	0.0e+00	0.31 / 3.01	4.5e-14	+0.080 / +0.064
2026-06-18/episode_002	right	0.0e+00	0.49 / 3.73	9.0e-14	+0.113 / +0.017
2026-06-18/episode_003	left	0.0e+00	0.29 / 2.35	4.3e-14	+0.107 / +0.101
2026-06-18/episode_003	right	0.0e+00	0.40 / 3.32	8.9e-14	+0.080 / +0.040

Across all episodes: free-space invariance 0.0e+00 m, roundtrip error 9.1e-14 N (machine precision), max penetration 15.2 mm.

What the numbers say

Method 1 works, with honest limits. Externally: ρ = 0.70 pooled (0.66–0.94 within an indenter family) against FEA ground truth on FEATS, in a transfer setting strictly harder than React's. Internally: SNR 0.7–6884184.8 across episodes and ρ = 0.18–0.95 vs the independent intensity proxy. Absolute scale is FEATS-calibrated; per-frame absolute error in the transfer test is ±5 N, so treat single-frame values as coarse and trends as reliable. Shear-dominant contact is a stated blind spot.
Method 2 is exactly as reliable as its input force. The transform itself is loss-free (invariance and roundtrip at machine precision) and keeps targets bounded (1.0 mm median, 15.2 mm at the single hardest ~23 N press — clamp or gain-schedule k in deployment if that exceeds the task tolerance). It converts pseudo-force into a deployable action without changing the policy's output space.
Recommended recipe: train on force-informed targets (Method 2) with the Method 1 force as an auxiliary prediction head — matching the conclusions of FARM and DexForce, adapted to markerless gel.

Debug log (what actually went wrong)

found	fix
FEATS returns its no-contact output on every markerless frame	switched force estimation to photometric-stereo depth (negative result kept above)
fixed dot-threshold 55 detects 0 dots (our dots bottom out at gray 56)	percentile-based threshold — then made moot by the markerless finding
1.1 mm phantom depth at the image border (Poisson/Neumann edge artifact)	exclude a 12/16-px margin from force integration
contact threshold exploded 30× when a reference frame was lightly touching	median zero-map + MAD threshold instead of mean + std
4% single-frame force spikes (bad Poisson solves)	median-3 over fresh frames only — row-wise filtering would see each duplicated value 3× and keep it
assumed gel normal [0,0,1] gave negative approach alignment at onsets	use the rig's dual-ball calibrated `gel_axis_in_rigid`; sign verified against approach kinematics
fed the full camera view to a depth MLP trained on the SDK's 15%-cropped view	same 1/7 border crop for React frames and FEATS images — geometry now identical between inference and validation
FEATS transfer started at ρ = 0.42: illumination tilt dwarfed real indentation, dot imprints leaked into depth	per-frame robust background-plane removal + marker inpainting → ρ = 0.70; quadratic background tried and rejected (absorbed contact)
shear-loaded captures anti-correlate (ρ = −0.15)	scoped the estimator to normal loading; documented as a limitation (React pushing involves shear — treat high-shear force estimates with caution)
theoretical Winkler scale off by ~40× (depth-unit + gel-constant assumptions)	absolute scale fitted on FEATS ground truth; React forces in FEATS-calibrated newtons, uncertainty stated

Data: yxma/React · code ships with the dataset (preprocess/, toolbox/) · references: FARM (2510.13324), DexForce (2501.10356), FEATS (2411.03315), ACP (2410.09309), gsrobotics SDK.