Spaces:
Running
Running
| <html lang="en"><head><meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width,initial-scale=1"> | |
| <title>Recovering Force Actions for the React Dataset</title> | |
| <style> | |
| :root { --accent:#d95f02; --accent2:#7570b3; --ok:#1b9e77; --bg:#faf9f7; | |
| --card:#ffffff; --text:#2b2b2b; --muted:#6b6b6b; } | |
| * { box-sizing:border-box; } | |
| body { font-family:'Segoe UI',system-ui,-apple-system,sans-serif; margin:0; | |
| background:var(--bg); color:var(--text); line-height:1.55; } | |
| .wrap { max-width:1060px; margin:0 auto; padding:0 20px 80px; } | |
| header { background:linear-gradient(135deg,#2d2a4a 0%,#4a3f6b 60%,#7a5c9e 100%); | |
| color:#f3f0ff; padding:52px 20px 44px; } | |
| header .wrap { padding-bottom:0; } | |
| header h1 { margin:0 0 10px; font-size:1.9rem; font-weight:650; } | |
| header p.sub { margin:0; max-width:820px; color:#d8d2ef; font-size:1.02rem; } | |
| header .pills { margin-top:18px; } | |
| .pill { display:inline-block; background:rgba(255,255,255,.14); padding:4px 12px; | |
| border-radius:999px; font-size:.82rem; margin-right:8px; } | |
| h2 { font-size:1.35rem; margin:52px 0 6px; } | |
| h3 { font-size:1.05rem; margin:26px 0 6px; } | |
| p.lead { color:var(--muted); margin-top:0; } | |
| .card { background:var(--card); border:1px solid #e8e4de; border-radius:12px; | |
| padding:20px 22px; margin:16px 0; box-shadow:0 1px 3px rgba(0,0,0,.04); } | |
| .card img, .card video { max-width:100%; border-radius:6px; display:block; | |
| margin:10px auto; } | |
| .negative { border-left:4px solid #c0392b; } | |
| .method { border-left:4px solid var(--accent); } | |
| .method2 { border-left:4px solid var(--accent2); } | |
| table { border-collapse:collapse; width:100%; font-size:.86rem; margin:10px 0; } | |
| th,td { padding:6px 9px; border-bottom:1px solid #eee7de; text-align:right; } | |
| th { background:#f4f0ea; font-weight:600; } | |
| td:first-child, th:first-child { text-align:left; } | |
| code { background:#f2eee8; padding:1px 5px; border-radius:4px; font-size:.86em; } | |
| .formula { text-align:center; font-family:Georgia,serif; font-style:italic; | |
| font-size:1.08rem; padding:12px; background:#f6f3ee; border-radius:8px; } | |
| .flow { display:flex; flex-wrap:wrap; gap:8px; align-items:center; | |
| justify-content:center; margin:14px 0; } | |
| .flow .step { background:#f0ebf7; border:1px solid #d9cfeb; padding:8px 14px; | |
| border-radius:8px; font-size:.85rem; text-align:center; } | |
| .flow .arr { color:var(--muted); font-size:1.1rem; } | |
| .verdict { font-weight:600; } | |
| .verdict.pass { color:var(--ok); } .verdict.warn { color:#c9820e; } | |
| .footnote { color:var(--muted); font-size:.82rem; } | |
| details { margin:10px 0; } | |
| details summary { cursor:pointer; color:var(--accent2); font-size:.85rem; | |
| font-weight:600; padding:8px 12px; background:#f4f0ea; border-radius:8px; } | |
| details[open] summary { border-radius:8px 8px 0 0; } | |
| .twocol { display:grid; grid-template-columns:1fr 1fr; gap:16px; } | |
| @media (max-width:760px){ .twocol { grid-template-columns:1fr; } | |
| header h1{font-size:1.4rem} | |
| table{display:block;overflow-x:auto;-webkit-overflow-scrolling:touch} | |
| th,td{min-width:100px} } | |
| </style></head><body> | |
| <header><div class="wrap"> | |
| <h1>Recovering Force-Related Actions for the React Tactile Dataset</h1> | |
| <p class="sub">React records GelSight-Mini images and OptiTrack poses β but, like | |
| all UMI-style data, no applied force: demonstrated pose equals achieved pose, so | |
| replaying the actions presses with zero intended force. Two methods put it back: | |
| tactile-estimated normal force, and DexForce-style force-informed action targets.</p> | |
| <div class="pills"><span class="pill">GelSight Mini (markerless gel)</span> | |
| <span class="pill">no F/T sensor</span><span class="pill">no force ground truth</span> | |
| <span class="pill">dataset: yxma/React</span> | |
| <a class="pill" href="method.html" style="text-decoration:none;color:#ffd9a0;border:1px solid rgba(255,255,255,.3)">β method design</a> | |
| <a class="pill" href="results.html" style="text-decoration:none;color:#ffd9a0;border:1px solid rgba(255,255,255,.3)">β results matrix</a> | |
| <a class="pill" href="actions.html" style="text-decoration:none;color:#ffd9a0;border:1px solid rgba(255,255,255,.3)">β action processing deep-dive</a></div> | |
| </div></header> | |
| <div class="wrap"> | |
| <h2>The negative result that shaped everything</h2> | |
| <div class="card negative"> | |
| <p style="margin-top:0"><b>FEATS (U-net, FEA-labeled) does not transfer:</b> it is | |
| trained on marker-dot gel, React's sensors are markerless β on our frames it | |
| returns its no-contact output regardless of contact. Verified in-domain at | |
| Ο = 0.96 on its own dataset, so this is a domain effect, not a weak model | |
| (<a href="results.html">full matrix</a>).</p> | |
| <img src="assets/feats_domain_gap.png" alt="training sensor vs our sensor"> | |
| <p class="footnote">Left: FEATS training gel (63 marker dots). Right: React gel, | |
| markerless.</p> | |
| </div> | |
| <h2>Method 1 β physics: depth reconstruction β normal force</h2> | |
| <div class="card method"> | |
| <div class="flow"> | |
| <span class="step">GelSight frame</span><span class="arr">β</span> | |
| <span class="step">dI = img β ref<br>(difference image)</span><span class="arr">β</span> | |
| <span class="step">per-sensor RGB LUT<br>β surface gradients</span><span class="arr">β</span> | |
| <span class="step">Poisson<br>integration</span><span class="arr">β</span> | |
| <span class="step">indentation Ξ΄</span><span class="arr">β</span> | |
| <span class="step">F(vol, area, max Ξ΄)</span> | |
| </div> | |
| <p>Zero training frames from our rig; the lookup table is self-calibrated from | |
| spherical presses (classic Dong/Yuan calibration, LUT-v2). Every design | |
| decision was driven by a measured defect β the full pipeline, | |
| the optimization journey, and the step-by-step reconstruction debug are on the | |
| <a href="method.html"><b>method page</b></a> and the | |
| <a href="debug_pipeline.html"><b>pipeline debug page</b></a>.</p> | |
| <img src="assets/depth_validation_panel.png" alt="raw | diff | depth"> | |
| <p class="footnote">Strongest motherboard presses: raw | difference | | |
| LUT-reconstructed depth. More examples (20 panels, 10 clips) in the | |
| <a href="gallery.html">gallery</a>.</p> | |
| </div> | |
| <h2>Validated against four force-labeled datasets</h2> | |
| <div class="card method"> | |
| <table> | |
| <tr><th>dataset (gel)</th><th>ours</th><th>FEATS U-net</th><th>FeelAnyForce</th></tr> | |
| <tr><td>FEATS val (marker)</td><td>0.77</td><td><b>0.96</b> in-domain</td><td>0.43</td></tr> | |
| <tr><td>FoTa cnc_Mini (markerless)</td><td>0.94 (in view)</td><td>0.07</td><td><b>0.83</b></td></tr> | |
| <tr><td>GlowTact (markerless)</td><td><b>0.98</b></td><td>0.04</td><td>0.90</td></tr> | |
| <tr><td>Sparsh / Meta (markerless, 10 pads)</td><td><b>0.97</b> *</td><td colspan="2">not run β no published predictions</td></tr> | |
| <tr><td>React (no GT β agreement)</td><td colspan="3">physics vs FeelAnyForce Ο = 0.91; both read β0 N off-contact</td></tr> | |
| </table> | |
| <p>Each network dominates its own gel domain and collapses | |
| outside it; the physics pipeline (0.74β0.99) is the only one that works | |
| everywhere. Predicted-vs-ground-truth scatters, per dataset, on the | |
| <a href="results.html"><b>results page</b></a>.</p> | |
| <p class="footnote" style="margin-bottom:0">* Sparsh is a <b>foreign sensor</b>. | |
| Our GlowTact table reaches 0.878 there; rebuilding the table from Sparsh's own | |
| sphere presses gives <b>0.968 (MAE 0.042 N)</b> and turns a bilobed | |
| reconstruction into a true dome. That is <b>calibrate once per sensor</b>, not | |
| zero-shot β and absolute newtons still do not transfer across indenter shapes. | |
| Details on the <a href="results.html">results page</a>.</p> | |
| </div> | |
| <h2>Method 2 β force-informed action targets</h2> | |
| <div class="card method2"> | |
| <p class="formula">p<sub>target</sub> = p<sub>observed</sub> + (FΜ<sub>n</sub> / k) Β· nΜ, | |
| k = 1500 N/m</p> | |
| <p>The estimated force becomes a virtual position target past the contact | |
| surface (DexForce-style): the action stays a pose, free space is untouched, and | |
| an impedance controller reproduces the demonstrated force at deployment. The | |
| transform is loss-free β invariance 0e+00 m, roundtrip 9e-14 N, | |
| penetration median 0.9 mm. Interactive walkthrough on the | |
| <a href="actions.html"><b>actions page</b></a>.</p> | |
| <img src="assets/dexforce_motherboard_episode_000_left.png" alt="virtual target offsets"> | |
| </div> | |
| <h2>Takeaway</h2> | |
| <div class="card"> | |
| <ul style="margin:0"> | |
| <li><b>Labelling React</b>: FeelAnyForce as primary labeller (strongest on | |
| markerless Mini), the physics pipeline as an independent audit β they agree at | |
| Ο = 0.91 and cannot share failure modes.</li> | |
| <li><b>Training</b>: use force-informed targets (Method 2) with the estimated | |
| force as an auxiliary head; absolute newtons are coarse (Β±25β40%), trends are | |
| reliable.</li> | |
| <li><b>New gel / new sensor</b>: the physics pipeline is the only estimator | |
| that works out of the box.</li> | |
| </ul> | |
| </div> | |
| <h2>Debug log</h2> | |
| <div class="card"> | |
| <details><summary>10 things that actually went wrong, and their fixes (click to expand)</summary> | |
| <table> | |
| <tr><th>found</th><th>fix</th></tr> | |
| <tr><td>FEATS returns its no-contact output on every markerless frame</td> | |
| <td>switched force estimation to photometric-stereo depth (negative result kept above)</td></tr> | |
| <tr><td>fixed dot-threshold 55 detects 0 dots (our dots bottom out at gray 56)</td> | |
| <td>percentile-based threshold β then made moot by the markerless finding</td></tr> | |
| <tr><td>1.1 mm phantom depth at the image border (Poisson/Neumann edge artifact)</td> | |
| <td>exclude a 12/16-px margin from force integration</td></tr> | |
| <tr><td>contact threshold exploded 30Γ when a reference frame was lightly touching</td> | |
| <td>median zero-map + MAD threshold instead of mean + std</td></tr> | |
| <tr><td>4% single-frame force spikes (bad Poisson solves)</td> | |
| <td>median-3 over fresh frames only β row-wise filtering would see each duplicated | |
| value 3Γ and keep it</td></tr> | |
| <tr><td>assumed gel normal [0,0,1] gave negative approach alignment at onsets</td> | |
| <td>use the rig's dual-ball calibrated <code>gel_axis_in_rigid</code>; sign verified | |
| against approach kinematics</td></tr> | |
| <tr><td>fed the full camera view to a depth MLP trained on the SDK's 15%-cropped view</td> | |
| <td>same 1/7 border crop for React frames and FEATS images β geometry now identical | |
| between inference and validation</td></tr> | |
| <tr><td>FEATS transfer started at Ο = 0.42: illumination tilt dwarfed real indentation, | |
| dot imprints leaked into depth</td> | |
| <td>per-frame robust background-plane removal + marker inpainting β Ο = 0.70; | |
| quadratic background tried and rejected (absorbed contact)</td></tr> | |
| <tr><td>shear-loaded captures anti-correlate (Ο = β0.15)</td> | |
| <td>scoped the estimator to normal loading; documented as a limitation (React pushing | |
| involves shear β treat high-shear force estimates with caution)</td></tr> | |
| <tr><td>theoretical Winkler scale off by ~40Γ (depth-unit + gel-constant assumptions)</td> | |
| <td>absolute scale fitted on FEATS ground truth; React forces in FEATS-calibrated | |
| newtons, uncertainty stated</td></tr> | |
| </table></details> | |
| </div> | |
| <p class="footnote">Data: <a href="https://huggingface.co/datasets/yxma/React"> | |
| yxma/React</a> Β· code ships with the dataset (<code>preprocess/</code>, | |
| <code>toolbox/</code>) Β· references: FARM (2510.13324), DexForce (2501.10356), | |
| FEATS (2411.03315), ACP (2410.09309), gsrobotics SDK.</p> | |
| </div></body></html> |