Spaces:
Running
Running
File size: 9,351 Bytes
dbc544f | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 | # Marionette Architecture
This document explains the design of the Marionette app for developers who want to understand, modify, or debug it.
## What Marionette Does
Marionette records and plays back head movements (+ optional audio) for the Reachy Mini robot. A user moves the robot's head by hand while the app records the pose at 100 Hz, then replays it as an animated movement. Audio can be recorded from the mic, uploaded as a file, downloaded from YouTube, or picked from the robot's filesystem.
## System Overview
```
┌─────────────────┐ HTTP (polling) ┌──────────────────┐
│ Browser (UI) │ ◄──────────────────────────── │ FastAPI Server │
│ main.js │ ──────────────────────────── ►│ (Uvicorn) │
│ index.html │ POST /api/record │ │
│ style.css │ POST /api/play │ marionette/ │
└─────────────────┘ GET /api/state │ ├── app.py │
│ ├── routes.py │
│ ├── recording.py│
│ ├── datasets.py │
│ ├── audio.py │
│ ├── state.py │
│ └── models.py │
└────────┬─────────┘
│
┌────────▼─────────┐
│ Reachy Mini │
│ (Robot SDK) │
└──────────────────┘
```
### Two-Machine vs Single-Machine
- **Reachy Mini**: Backend runs on the robot (Linux ARM), browser runs on a laptop. Connected via WiFi.
- **Reachy Mini Light**: Backend and browser run on the same laptop. The robot connects via USB.
## Module Map
| Module | Purpose | Key classes/functions |
|--------|---------|---------------------|
| `models.py` | Data types, constants, Pydantic models | `RecordingMetadata`, `RecordingRequest`, `DatasetEntry`, all `*Payload` classes |
| `audio.py` | Stateless audio functions | `play_wav_chunked()`, `preload_wav()`, `play_preloaded_wav()` |
| `state.py` | Thread-safe state read/write | `StateMixin._serialize_state()`, `_set_state()`, `_set_idle_state()` |
| `recording.py` | Motion capture + playback | `RecordingMixin._capture_motion()`, `_perform_recording()`, `_perform_playback()` |
| `datasets.py` | Dataset filesystem + HF sync | `DatasetMixin._load_dataset_registry()`, `_sync_dataset()`, `_check_hf_login()` |
| `routes.py` | HTTP endpoint definitions | `register_routes()` — all FastAPI route closures |
| `app.py` | Main class, run loop, robot helpers | `Marionette`, `create_app()` |
| `main.py` | Re-export hub | Imports and re-exports everything for backward compatibility |
| `motion_models.py` | Lead compensation model | `MotionModelRegistry`, shifts commands forward to counter mechanical lag |
## Threading Model
The app has three types of threads:
1. **Uvicorn thread** — Runs the FastAPI HTTP server. Handles all API requests. This is the thread that calls route handlers in `routes.py`.
2. **Main robot thread** — Runs `Marionette.run()`. Polls for pending jobs (recording or playback) in a 50ms loop. Executes recording/playback synchronously.
3. **Audio threads** — Spawned as daemon threads during playback. Push audio chunks to the robot's GStreamer pipeline.
### Thread Safety
All shared state is protected by `_state_lock` (a `threading.Lock`):
- The HTTP thread writes: `_pending_recording`, `_pending_playback`, `_mode`
- The robot thread reads and clears: `_pending_recording`, `_pending_playback`
- Both threads read: `_mode`, `_recordings`, `_datasets`
The lock is held briefly — never during I/O or network calls.
### Cancel Events
- `_recording_cancel_event` — Set by the HTTP thread (POST /api/record/stop), checked by the robot thread's capture loop.
- `_playback_cancel_event` — Set by the HTTP thread (POST /api/play/stop), checked by the robot thread's playback loop.
## State Machine
```
POST /api/record
idle ──────────────────── queued
▲ │
│ (robot thread picks up)
│ ▼
│ countdown (3s)
│ │
│ ▼
└──────────────────── recording
│
(duration elapsed or stop)
▼
idle
POST /api/play
idle ──────────────────── queued
▲ │
│ (robot thread picks up)
│ ▼
└──────────────────── playing
│
(move ends or stop)
▼
idle
```
Additional states: `starting_up` (during boot animation), `error` (transient, returns to idle).
## Frontend Architecture
The frontend is vanilla JavaScript (no framework). Key patterns:
### Polling Loop
The browser polls `GET /api/state` at regular intervals:
- **1500ms** when idle (nothing happening)
- **200ms** when active (recording, playing, countdown)
Each poll returns the full app state. The frontend updates the UI accordingly.
### Dirty Flag Pattern (Flickering Fix)
The moves list and dataset dropdown are expensive to rebuild (full DOM replacement). Without optimization, they flicker on every poll. The fix:
- `movesListDirty` and `datasetsDirty` flags start as `true`
- `renderMoves()` / `updateDatasetUI()` only run when the flag is `true`
- Flags are set `true` after user actions (record, delete, switch dataset, etc.)
- During idle polling, only lightweight updates happen (mode badge, play/stop buttons)
### NTP-Style Clock Sync
The backend sends `server_time` with each state response. The frontend computes `clockOffset = serverTime - localTime` and uses it to accurately display countdown timers and recording progress bars, even when the browser and robot are on different machines with unsynchronized clocks.
### Phase Overlay
During countdown and recording, a full-screen overlay shows progress. This uses `requestAnimationFrame` for smooth 60fps animation, independent of the polling interval.
## Dataset Layout
```
~/reachy_mini_datasets/ # dataset root (configurable)
├── local_dataset/ # default dataset
│ └── data/ # all recordings live here
│ ├── happy-dance.json # motion trajectory
│ └── happy-dance.wav # optional audio
├── my-custom-set/
│ └── data/
│ └── ...
└── user-community-set/ # downloaded from HF
└── data/
└── ...
```
The `dataset_registry.json` file (next to the app) tracks which datasets exist, which is active, and per-dataset metadata (uploaded move IDs, origin, etc.).
## Audio Playback Pipeline
Audio is played through the Reachy Mini's GStreamer pipeline using chunk-based pushing:
1. **Preload**: Read WAV file, resample to output rate if needed
2. **Prime pipeline**: Call `start_playing()` to initialize GStreamer
3. **Wait for sync**: Audio thread waits for `start_signal` (set when first motion command is sent)
4. **Push chunks**: Feed 20ms audio chunks at ~1.25x real-time
5. **Drain buffer**: Wait for remaining audio to play out
6. **Cleanup**: Call `stop_playing()` with timeout (GStreamer can hang)
This push-based approach allows stopping audio at any time, unlike `play_sound()` which creates an uninterruptible pipeline.
## Lead Compensation
Mechanical lag means the robot's actual motion trails the commanded trajectory. The lead compensation model (in `motion_models.py`) shifts commands forward in time so the actual motion matches the original recording. Parameters (head lead, antenna lead) are tunable in the settings UI.
## Testing
Tests live in `tests/test_api.py` and use FastAPI's `TestClient` for synchronous HTTP testing without a real robot. The `conftest.py` creates a Marionette instance with a temporary dataset directory.
Key test patterns:
- HF functions are monkeypatched on `marionette.datasets` (the canonical location)
- Audio functions are monkeypatched on `marionette.recording` (where they're imported)
- Recording/playback tests use fake `ReachyMini` objects with stub methods
|