Spaces:
Running
Running
| <html lang="en"><head><meta charset="utf-8"> | |
| <meta name="viewport" content="width=device-width,initial-scale=1"> | |
| <title>Method in One Page — React Force Recovery</title> | |
| <link rel="preconnect" href="https://fonts.googleapis.com"> | |
| <link href="https://fonts.googleapis.com/css2?family=Fraunces:opsz,wght@9..144,600;9..144,700&family=IBM+Plex+Mono:wght@400;500&family=IBM+Plex+Sans:wght@400;600&family=Noto+Serif+SC:wght@600;700&family=Noto+Sans+SC:wght@400;500&display=swap" rel="stylesheet"> | |
| <style> | |
| :root{--paper:#0d1526;--ink:#dce7f5;--dim:#7c8db0;--grid:#1a2540; | |
| --force:#ffb347;--target:#ff7847;--ok:#7be0a0;--card:#111b31;--line:#24345;} | |
| *{box-sizing:border-box} | |
| body{margin:0;background:var(--paper);color:var(--ink); | |
| font-family:'IBM Plex Sans','Noto Sans SC',sans-serif;line-height:1.65;font-size:16px; | |
| background-image:linear-gradient(var(--grid) 1px,transparent 1px), | |
| linear-gradient(90deg,var(--grid) 1px,transparent 1px);background-size:44px 44px} | |
| .wrap{max-width:880px;margin:0 auto;padding:0 24px 80px} | |
| h1,h2{font-family:'Fraunces','Noto Serif SC',serif} | |
| h1{font-size:2.1rem;font-weight:700;line-height:1.2;margin:0 0 10px} | |
| h2{font-size:1.25rem;font-weight:600;margin:46px 0 8px;color:var(--force)} | |
| .kicker{font-family:'IBM Plex Mono',monospace;font-size:.76rem;letter-spacing:.16em; | |
| text-transform:uppercase;color:var(--force);margin-bottom:14px} | |
| header{padding:60px 0 26px;border-bottom:1px dashed var(--line)} | |
| .sub{color:var(--dim);max-width:720px} | |
| .card{background:var(--card);border:1px solid var(--line);border-radius:10px; | |
| padding:18px 22px;margin:14px 0} | |
| .step{display:grid;grid-template-columns:34px 200px 1fr;gap:12px; | |
| padding:10px 0;border-bottom:1px solid var(--line);align-items:baseline} | |
| .step:last-child{border-bottom:none} | |
| .step .n{font-family:'Fraunces',serif;font-size:1.25rem;color:var(--target)} | |
| .step .t{font-family:'IBM Plex Mono',monospace;font-size:.85rem} | |
| .step .d{color:var(--dim);font-size:.9rem} | |
| table{border-collapse:collapse;width:100%;font-size:.88rem;margin:10px 0; | |
| font-variant-numeric:tabular-nums} | |
| th,td{padding:7px 10px;border-bottom:1px solid var(--line);text-align:right} | |
| th{color:var(--dim);font-family:'IBM Plex Mono',monospace;font-size:.72rem; | |
| text-transform:uppercase;letter-spacing:.08em} | |
| td:first-child,th:first-child{text-align:left} | |
| img{max-width:100%;border-radius:8px;border:1px solid var(--line);display:block;margin:12px auto} | |
| a{color:#4fd8e0}a:hover{color:var(--force)} | |
| .pill{display:inline-block;border:1px solid var(--line);border-radius:999px; | |
| padding:3px 12px;font-size:.75rem;font-family:'IBM Plex Mono',monospace; | |
| color:var(--dim);margin-right:8px;margin-top:12px;text-decoration:none} | |
| code{font-family:'IBM Plex Mono',monospace;background:#0a1120;padding:1px 5px; | |
| border-radius:4px;font-size:.85em} | |
| .ok{color:var(--ok)}.warn{color:#e0b34f} | |
| footer{margin-top:60px;padding-top:18px;border-top:1px dashed var(--line); | |
| color:var(--dim);font-size:.8rem} | |
| @media(max-width:640px){.step{grid-template-columns:28px 1fr;grid-auto-flow:dense} | |
| .step .d{grid-column:2} | |
| h1{font-size:1.45rem} | |
| table{display:block;overflow-x:auto;white-space:nowrap;-webkit-overflow-scrolling:touch} | |
| th,td{white-space:normal;min-width:110px}} | |
| </style></head><body><div class="wrap"> | |
| <header> | |
| <div class="kicker">React force recovery · method overview</div> | |
| <h1>GelSight image → normal force, in one page</h1> | |
| <p class="sub">Markerless gel, no F/T sensor, no training data from our rig. | |
| A physics pipeline with exactly one fitted number.</p> | |
| <a class="pill" href="results.html">↖ results matrix</a> | |
| <a class="pill" href="index.html">overview</a> | |
| <a class="pill" href="actions.html">action transform</a> | |
| <a class="pill" href="method_zh.html">中文</a> | |
| </header> | |
| <h2>The pipeline — six steps</h2> | |
| <div class="card"> | |
| <div class="step"><span class="n">1</span><span class="t">crop 1/7 border</span> | |
| <span class="d">the depth network was trained on the SDK's cropped view; the full frame includes LED borders it has never seen</span></div> | |
| <div class="step"><span class="n">2</span><span class="t">RGB → surface normals</span> | |
| <span class="d">per-pixel MLP (gsrobotics <code>nnmini</code>): three-color illumination makes color→normal invertible</span></div> | |
| <div class="step"><span class="n">3</span><span class="t">Poisson integration</span> | |
| <span class="d">normals → height map; subtract a per-episode zero map (median of the 15 lowest-contact frames)</span></div> | |
| <div class="step"><span class="n">4</span><span class="t">background plane removal</span> | |
| <span class="d">illumination drift integrates into a global tilt that can dwarf real indentation; a robust per-frame plane fit removes it</span></div> | |
| <div class="step"><span class="n">5</span><span class="t">contact threshold</span> | |
| <span class="d">5σ from the MAD of reference-frame residuals — per sensor, because noise varies 10–50 µm between sensors</span></div> | |
| <div class="step"><span class="n">6</span><span class="t">volume × c → force</span> | |
| <span class="d">Winkler foundation: F = c·Σδ·dA. The scale c is the single fitted number — from FEA ground truth, not assumed gel constants</span></div> | |
| </div> | |
| <p>Post-processing: a 3-tap median over <i>fresh</i> tactile frames only | |
| (duplicated rows would let a row-wise filter count bad values three times). | |
| Cuts single-frame spikes from 4–8% to ≈0.</p> | |
| <h2>Photometric overhaul — classic per-sensor calibration (v2)</h2> | |
| <div class="card"> | |
| <p style="margin-top:0">Ground-truth depth supervision (commanded press depth in | |
| GlowTact) exposed that the generic depth MLP recovers only ~25% of true | |
| indentation and saturates (peak-depth ρ = 0.39) — and, deeper, that the | |
| <b>gsrobotics SDK's Poisson solver returns 39% of the amplitude even on a | |
| perfect synthetic gradient field</b> (line-integral proved the gradients were | |
| correct at 105%). Rebuilt on the classic Dong/Yuan calibration: difference | |
| image → per-sensor RGB lookup table, self-calibrated from GlowTact's | |
| spherical presses via the exact relation a² = d(2R−d) (R = 3.35 mm, no | |
| external data) → exact Poisson (Dong's fast_poisson, 100.7% on the same | |
| benchmark) → sphere-supervised spatial gain field → Drake-style stiffening | |
| foundation p = k₁δ + k₂δ² with imprint-derived shape conditioning.</p> | |
| <table> | |
| <tr><th>stage (held-out, GlowTact 0–20 N)</th><th>ρ</th><th>MAE</th></tr> | |
| <tr><td>MLP + linear Winkler (v1)</td><td>0.63</td><td>4.4 N</td></tr> | |
| <tr><td>LUT + solver fix + gain field + nonlinear foundation</td><td>0.80</td><td>2.75 N</td></tr> | |
| <tr><td>+ imprint shape self-conditioning</td><td><b>0.82</b></td><td><b>2.46 N</b></td></tr> | |
| <tr><td>spheres only (geometry exact — the method's ceiling)</td><td class="ok"><b>0.91–0.94</b></td><td class="ok">1.5–1.9 N</td></tr> | |
| <tr><td><b>spheres × 0–8 N</b> (React's operating range; 7-seed median, +isotonic)</td><td class="ok"><b>0.95</b> (0.93–0.96)</td><td class="ok"><b>0.78 N</b> (0.73–0.84)</td></tr> | |
| </table> | |
| <p class="footnote">Ceiling context: the CNC's own commanded depth predicts force | |
| at ρ = 0.975. The remaining pooled gap is object-dependent contact mechanics; | |
| sub-newton MAE on 0–20 N exceeds what geometry alone carries (the 200K-frame | |
| supervised network reaches 2.1 N on the same range). A new sensor needs one | |
| 2-minute ball-press pass — a calibration the React rig can adopt.</p> | |
| </div> | |
| <h2>Validation (v1 pipeline)</h2> | |
| <div class="card"> | |
| <p style="margin-top:0">The v1 pipeline is validated on <b>three force-labeled | |
| datasets</b> (FEATS, FoTa cnc_Mini, GlowTact) and cross-checked against two | |
| neural estimators on identical frames — every predicted-vs-ground-truth | |
| scatter, per dataset, lives on the | |
| <a href="results.html"><b>results page</b></a>. Short version: physics | |
| 0.43-0.74 everywhere; each network 0.90+ in its own gel domain and collapsing | |
| outside it.</p> | |
| </div> | |
| <h2>Optimizing against ground truth</h2> | |
| <div class="card"> | |
| <p style="margin-top:0">The cnc_Mini force labels turned the pipeline's weak spots into | |
| measurable defects, fixed in order (each step verified on held-out data):</p> | |
| <table> | |
| <tr><th>step</th><th>evidence that drove it</th><th>ρ (held-out val)</th></tr> | |
| <tr><td>baseline (volume, per-episode zeroing)</td><td>—</td><td>0.34 pooled</td></tr> | |
| <tr><td>+ median zero map over scattered presses</td><td>only 4 of 2686 frames are truly contact-free; lowest-force references carried 1.7 N of baked-in contact</td><td>≈ same (zero map wasn't the bottleneck)</td></tr> | |
| <tr><td>+ <b>flat-field illumination normalization</b></td><td>force-fit residuals correlated with contact position (|ρ| up to 0.6); probe×quadrant conditioning raised ρ 0.45→0.55 — the vignette modulates the depth MLP's gain</td><td>0.44 pooled</td></tr> | |
| <tr><td>+ <b>edge filtering</b> (contact centroid > 3 mm from border)</td><td>border presses sit where the vignette is steepest and imprints clip the sensor edge</td><td><b>0.65</b> (probe-median 0.64)</td></tr> | |
| </table> | |
| <p><b>Volume or max depth?</b> Settled empirically: the volume family wins | |
| (vol<sup>1.5</sup>-weighted 0.65, plain volume 0.63) over max depth (0.63) and | |
| clearly over contact area (0.35); a tiny 3-feature linear model matches ρ but | |
| halves MAE (0.61 N). The ceiling matters too: even the CNC's own commanded | |
| press depth only reaches ρ 0.78–0.88 <i>within</i> a probe — force at equal | |
| depth genuinely varies with texture and position.</p> | |
| </div> | |
| <h2>Gallery</h2> | |
| <div class="card"> | |
| <p style="margin-top:0">20 image samples (raw | indentation | 3D reconstruction | | |
| predicted vs ground-truth force) and 10 React episode clips | |
| (tactile | live depth | force trace): <a href="gallery.html">browse the full | |
| gallery</a>.</p> | |
| <img src="assets/gallery/cnc_08.png" alt="sample panel"> | |
| <video controls muted loop playsinline preload="metadata" | |
| src="assets/gallery/clip_motherboard_episode_000_left.mp4"></video> | |
| </div> | |
| <h2>NN vs model-based on the GelSight Mini — who has compared them?</h2> | |
| <div class="card"> | |
| <p style="margin-top:0"><b>No published head-to-head that we could find.</b> | |
| The literature runs in two camps that cite but don't benchmark each other. | |
| Model-based: marker displacement × elasticity | |
| (<a href="https://ieeexplore.ieee.org/document/8202149">Yuan 2017</a>), | |
| photometric-stereo height + polynomial fit, inverse FEM | |
| (<a href="https://arxiv.org/abs/1810.04621">GelSlim, Ma 2019</a>). | |
| Learned, on the Mini specifically: | |
| <a href="https://openreview.net/forum?id=dUO0QQw4FW">CANFnet</a> (F/T-labeled, normal only), | |
| <a href="https://arxiv.org/abs/2411.03315">FEATS</a> (FEA-labeled, 3D distributions), | |
| <a href="https://arxiv.org/abs/2410.02048">FeelAnyForce</a> (200K ATI-labeled). | |
| Each motivates NN over physics qualitatively — FEA too slow for real time, | |
| linear elasticity misses elastomer nonlinearity — but their reported baselines | |
| are other <i>networks</i>, not the physics pipeline.</p> | |
| <p>Our FEATS experiment is therefore one of the few direct data points: | |
| the model-based pipeline reaches ρ 0.70–0.85 with <b>one</b> fitted scalar and | |
| zero training frames, where the NNs earn sub-newton MAE in-domain but die | |
| outside their gel (FEATS on our markerless gel: no response at all). | |
| The trade is portability vs in-domain accuracy — and which one you need | |
| depends on whether you can collect labels on your own sensor.</p> | |
| </div> | |
| <footer>React force recovery · | |
| <a href="https://huggingface.co/datasets/yxma/React">dataset</a> · | |
| <a href="index.html">results</a> · | |
| <a href="actions.html">action transform</a> · | |
| code: <code>twm/force_recovery/</code></footer> | |
| </div></body></html> |