# Marionette Architecture This document explains the design of the Marionette app for developers who want to understand, modify, or debug it. ## What Marionette Does Marionette records and plays back head movements (+ optional audio) for the Reachy Mini robot. A user moves the robot's head by hand while the app records the pose at 100 Hz, then replays it as an animated movement. Audio can be recorded from the mic, uploaded as a file, downloaded from YouTube, or picked from the robot's filesystem. ## System Overview ``` ┌─────────────────┐ HTTP (polling) ┌──────────────────┐ │ Browser (UI) │ ◄──────────────────────────── │ FastAPI Server │ │ main.js │ ──────────────────────────── ►│ (Uvicorn) │ │ index.html │ POST /api/record │ │ │ style.css │ POST /api/play │ marionette/ │ └─────────────────┘ GET /api/state │ ├── app.py │ │ ├── routes.py │ │ ├── recording.py│ │ ├── datasets.py │ │ ├── audio.py │ │ ├── state.py │ │ └── models.py │ └────────┬─────────┘ │ ┌────────▼─────────┐ │ Reachy Mini │ │ (Robot SDK) │ └──────────────────┘ ``` ### Two-Machine vs Single-Machine - **Reachy Mini**: Backend runs on the robot (Linux ARM), browser runs on a laptop. Connected via WiFi. - **Reachy Mini Light**: Backend and browser run on the same laptop. The robot connects via USB. ## Module Map | Module | Purpose | Key classes/functions | |--------|---------|---------------------| | `models.py` | Data types, constants, Pydantic models | `RecordingMetadata`, `RecordingRequest`, `DatasetEntry`, all `*Payload` classes | | `audio.py` | Stateless audio functions | `play_wav_chunked()`, `preload_wav()`, `play_preloaded_wav()` | | `state.py` | Thread-safe state read/write | `StateMixin._serialize_state()`, `_set_state()`, `_set_idle_state()` | | `recording.py` | Motion capture + playback | `RecordingMixin._capture_motion()`, `_perform_recording()`, `_perform_playback()` | | `datasets.py` | Dataset filesystem + HF sync | `DatasetMixin._load_dataset_registry()`, `_sync_dataset()`, `_check_hf_login()` | | `routes.py` | HTTP endpoint definitions | `register_routes()` — all FastAPI route closures | | `app.py` | Main class, run loop, robot helpers | `Marionette`, `create_app()` | | `main.py` | Re-export hub | Imports and re-exports everything for backward compatibility | | `motion_models.py` | Lead compensation model | `MotionModelRegistry`, shifts commands forward to counter mechanical lag | ## Threading Model The app has three types of threads: 1. **Uvicorn thread** — Runs the FastAPI HTTP server. Handles all API requests. This is the thread that calls route handlers in `routes.py`. 2. **Main robot thread** — Runs `Marionette.run()`. Polls for pending jobs (recording or playback) in a 50ms loop. Executes recording/playback synchronously. 3. **Audio threads** — Spawned as daemon threads during playback. Push audio chunks to the robot's GStreamer pipeline. ### Thread Safety All shared state is protected by `_state_lock` (a `threading.Lock`): - The HTTP thread writes: `_pending_recording`, `_pending_playback`, `_mode` - The robot thread reads and clears: `_pending_recording`, `_pending_playback` - Both threads read: `_mode`, `_recordings`, `_datasets` The lock is held briefly — never during I/O or network calls. ### Cancel Events - `_recording_cancel_event` — Set by the HTTP thread (POST /api/record/stop), checked by the robot thread's capture loop. - `_playback_cancel_event` — Set by the HTTP thread (POST /api/play/stop), checked by the robot thread's playback loop. ## State Machine ``` POST /api/record idle ──────────────────── queued ▲ │ │ (robot thread picks up) │ ▼ │ countdown (3s) │ │ │ ▼ └──────────────────── recording │ (duration elapsed or stop) ▼ idle POST /api/play idle ──────────────────── queued ▲ │ │ (robot thread picks up) │ ▼ └──────────────────── playing │ (move ends or stop) ▼ idle ``` Additional states: `starting_up` (during boot animation), `error` (transient, returns to idle). ## Frontend Architecture The frontend is vanilla JavaScript (no framework). Key patterns: ### Polling Loop The browser polls `GET /api/state` at regular intervals: - **1500ms** when idle (nothing happening) - **200ms** when active (recording, playing, countdown) Each poll returns the full app state. The frontend updates the UI accordingly. ### Dirty Flag Pattern (Flickering Fix) The moves list and dataset dropdown are expensive to rebuild (full DOM replacement). Without optimization, they flicker on every poll. The fix: - `movesListDirty` and `datasetsDirty` flags start as `true` - `renderMoves()` / `updateDatasetUI()` only run when the flag is `true` - Flags are set `true` after user actions (record, delete, switch dataset, etc.) - During idle polling, only lightweight updates happen (mode badge, play/stop buttons) ### NTP-Style Clock Sync The backend sends `server_time` with each state response. The frontend computes `clockOffset = serverTime - localTime` and uses it to accurately display countdown timers and recording progress bars, even when the browser and robot are on different machines with unsynchronized clocks. ### Phase Overlay During countdown and recording, a full-screen overlay shows progress. This uses `requestAnimationFrame` for smooth 60fps animation, independent of the polling interval. ## Dataset Layout ``` ~/reachy_mini_datasets/ # dataset root (configurable) ├── local_dataset/ # default dataset │ └── data/ # all recordings live here │ ├── happy-dance.json # motion trajectory │ └── happy-dance.wav # optional audio ├── my-custom-set/ │ └── data/ │ └── ... └── user-community-set/ # downloaded from HF └── data/ └── ... ``` The `dataset_registry.json` file (next to the app) tracks which datasets exist, which is active, and per-dataset metadata (uploaded move IDs, origin, etc.). ## Audio Playback Pipeline Audio is played through the Reachy Mini's GStreamer pipeline using chunk-based pushing: 1. **Preload**: Read WAV file, resample to output rate if needed 2. **Prime pipeline**: Call `start_playing()` to initialize GStreamer 3. **Wait for sync**: Audio thread waits for `start_signal` (set when first motion command is sent) 4. **Push chunks**: Feed 20ms audio chunks at ~1.25x real-time 5. **Drain buffer**: Wait for remaining audio to play out 6. **Cleanup**: Call `stop_playing()` with timeout (GStreamer can hang) This push-based approach allows stopping audio at any time, unlike `play_sound()` which creates an uninterruptible pipeline. ## Lead Compensation Mechanical lag means the robot's actual motion trails the commanded trajectory. The lead compensation model (in `motion_models.py`) shifts commands forward in time so the actual motion matches the original recording. Parameters (head lead, antenna lead) are tunable in the settings UI. ## Testing Tests live in `tests/test_api.py` and use FastAPI's `TestClient` for synchronous HTTP testing without a real robot. The `conftest.py` creates a Marionette instance with a temporary dataset directory. Key test patterns: - HF functions are monkeypatched on `marionette.datasets` (the canonical location) - Audio functions are monkeypatched on `marionette.recording` (where they're imported) - Recording/playback tests use fake `ReachyMini` objects with stub methods